Posted: 31 Dec 2020 15:36 EST Last activity: 7 Jan 2021 1:21 EST
Application architecture is complex: How to configure the pipelines - an approach
Deployment Manager is a great tool to configure a pipeline that deploys your implementation from one environment to another, from development environment to test environment to acceptance test environment and to production.
When your implementation is small and simple you don’t need to think a lot about the pipeline configuration in Deployment Manager: you create a new pipeline and Deployment Manager brings on a default configuration that you can use right away, no or only little adjustments are required.
But what if your implementation is not small and simple? What if you have several implementations that are built on a common shared layer which is built on a common enterprise layer?
What if your implementations need to support localizations in several languages? What if you get the translations from a specialized team that provides the localizations at a different pace than you implement corresponding rules? What if you work in a sprint but localizations are provided only later?
What if you cannot deploy immediately to higher environments because you have to wait until it’s time to deploy?
What if you have multiple teams modifying rules in the common shared layer or in the enterprise layer? How do ensure that changes made by one team don’t impact rules implemented by another team?
How can you ensure that test cases are run properly? When and in which environment should they run?
We were faced with questions like this and we took an approach that helped us a lot in answering and solving those questions.
I’d like to share this approach. Maybe it sparks a discussion. Maybe it gives you ideas for your own pipeline configurations.
Let’s start with the application architecture. The goal is to create several implementations. They use a common shared layer as many processes can be re-used. And we have the enterprise layer at the bottom:
Each implementation is supposed to support up to 5 different languages.
As usual, we added a “development” application on top of each deliverable application in which we create branches, it also contains the test rulesets for all layers. And it finally has a dev ruleset for sandbox rules we do not plan to deploy. For simplicity reasons, I’m considering only MyAppl1 from now on and will refer to it as MyAppl:
We decided to de-couple continuous integration from continuous deployment. Since there are several independent streams implementing their rules, we wanted to be able to test early in the Dev environment before we deploy to the next environment.
That means we have dedicated pipelines for CI and for CD:
MyAppl_CI is the Continuous Integration pipeline that merges branches into the trunk. It merges rules belonging to any of the three layers: MyAppl, MyCoShared and MyCo.
MyAppl_CD is the Continuous Deployment pipeline that deploys the MyAppl layer (and only this layer) to higher environments.
MyCoShared_CD and MyCo_CD are the Continous Deployment pipelines deploying the MyCoShared layer and MyCo layer resp. to higher environment.
MyAppl_CI is triggered from DevStudio when a branch is merged.
The CD pipelines need to be triggered manually from the Deployment Manager portal.
The CI pipeline
MyAppl_CI is configured to merge only.
The stages Quality Assurance, Staging and Production are not checked, therefor not executed. Only the steps in the Development stage are executed.
In the first pillar we check that there are no merge conflicts and that guardrail score of the merged branch is sufficient. Then we merge and trigger the second pillar.
In the second pillar we run a test suite that runs test cases across all layer to ensure that the merged rules don’t have any negative impacts. The test approach is discussed a bit further below.
The configuration of the pipeline settings is fairly simple: It uses MyApplDev as application, MyApplDev:Administrator as access group and MyApplDev as product rule. The MyApplDev product rule is empty, since we merge only in this pipeline, it’s only needed to populate the mandatory product rule field in the pipeline settings.
The CD pipelines
The CD pipelines are configured to deploy to higher environments. For each of the layers we have one CD pipeline. This is the MyAppl_CD pipeline, the other CD pipelines are configured analogously:
By default, we deploy only to Quality Assurance on a regular basis, and only on demand to Staging and Production. That’s why Quality Assurance is checked and Staging and Production are unchecked. When we want to deploy to the higher stages, we have to tick the corresponding stage. On top of that, we added a manual step “Deploy to Staging?” and “Deploy to Production?” respectively. In case, we forgot to uncheck, we still have a chance to prevent a deployment to the higher stages when the manual step is reached.
In this pipeline we basically verify that the guardrail score is still high, and we run a test suite that is specific to this layer to ensure that we still don’t have any negative impacts from newly merged rules in this layer. Please see also the test approach a little further below.
In the Quality Assurance stage we also trigger another pipeline: the localization pipeline which deploys only localizations. Please see a little further below for more information.
The configuration of the pipeline settings uses MyAppl as application, MyAppl:Administrators as access group and the MyAppl product rule. This product rule is configured to pack all rules belonging to this layer into the deployment file, mainly the MyAppl and MyApplInt rulesets.
This pipeline is deploying test cases for this layer, it uses MyApplTest as application, MyApplTest:Administrators as access group and MyApplTest as product rule.
Application MyApplTest is built on MyAppl and contains the rulesets MyApplTest, MyCoSharedTest and MyCoTest.
Product MyApplTest packs the rules of ruleset MyApplTest.
The pipelines MyCoShared_CD and MyCo_CD are using a similar configuration: they have their own applications, access groups and products according to their layer.
In order to simplify the configuration, we re-use the MyApplTest application and MyApplTest:Administrators access group in all three CD pipelines.
The Localization pipelines
In each CD pipeline we trigger a Localization pipeline, for example in MyAppl_CD we trigger pipeline MyAppl_Localization_CD
In each layer we have such a Localization pipeline. The configuration is simple: It uses MyAppl as application, MyAppl:Administrators as access groups and MyApplLocalization as product rule. The configuration of product MyApplLocalization is again quite simple: it packs all localization rulesets for this layer.
MyCoShared_Localization_CD and MyCo_Localization_CD are configured analogously, they use their own applications, access groups and product rules.
Running Test Suites
Our test approach is based on the run of test suites. We run test suites in the CI pipeline when we merge a branch, and in the CD pipelines when we deploy to higher environments.
Each CD pipeline runs a test suite that only contains test cases belonging to this layer. So we have three main test suites:
TS_MyAppl (needs to be specified as MYCO-MYAPPL-WORK!TS_MYAPPL in the Run Pega unit tests step)
TS_MyCoShared (needs to be specified as MYCO-MYAPPL-WORK!TS_MYCOSHARED in the Run Pega unit tests step)
TS_MyCo (needs to be specified as MYCO-MYAPPL-WORK!TS_MYCO in the Run Pega unit tests step)
Use access group MyApplTest:Administrators in this step in all three pipelines.
TS_MyAppl is a test suite that calls test cases and other test suites that again call test cases and other test suites, and so on. You can group test cases in test suites or you can put all test cases into the main test suite, whatever matches your needs best.
When you create test cases you must not forget to put each test case into a test suite, otherwise it won’t be called.
In the CI pipeline, we also call a test suite:
This test suite is meant to ensure quality across all layers. This is achieved quite easily: we run a test suite called TS_MyAppl_CI. TS_MyAppl_CI simply calls the other three test suites: TS_MyAppl, TS_MyCoShared and TS_MyCo.
The whole picture
The full set of pipelines:
***Edited by Moderator Marissa to add the Developer Knowledge Share tag***
many thanks for your questions, actually good points they are.
As for running the localization pipelines. We added this step in the CD pipelines as we usually have localized field values to deploy. It's part of our Definition of Ready to urge Business to provide translations and part of our Definition of Done to implement them. Reality shows, they're not always able to do so, that's why we sometimes implement first translations done by local consultants. What I'm trying to say is: we usually have localizations to deploy, that's why we added the step in the CD pipeline. But sometimes we get better translations a little later, and we can deploy them independently by manually running the Localization pipeline when needed.
As for identifying the localization rulesets: Yes, it's configured in a product rule. Each pipeline requires a product rule and will deploy the artifact (basically the generated product file) as a result of this product rule. That's why the Localization pipelines have their own Localization product rule. You just add the localization rulesets (ending with _fr, _de, _es, ...) to that product rule.
Posted: 2 weeks ago
Updated: 2 weeks ago
Posted: 7 Jan 2021 1:17 EST Updated: 7 Jan 2021 1:21 EST
Thank you so much for sharing detailed document on CICD.
Couple of questions,
What is the devOps version? we are on 7.4 I see Trigger deployment step is not available.
I believe we need to start deployment manually from MyAppl_CD pipeline? or does it triggered from MyApplDev.
What will happen to the deployment in MyAppl_CD pipeline? does it complete or stay as active deployment? because no further steps specified after trigger deployment step.
When its time to deploy to prod from where does the deployment file takes from?
When there is an active deployment, is it possible to update Pipeline to check Stage or prod release?
If we have multiple releases deployed until QA environment for example, MyApp:01.01.01 product rule with changes in RSV 01-01-01, MyApp:01.01.02 product rule with changes in RSV 01-01-02 and so on until MyApp:01.01.05. Now its time to release all 5 deployment files, how to achieve?