Speeding up AEM Cloud Service Deployment: The Dispatcher-Only Pipeline

Adobe just introduced a new feature for Cloud Manager, their CI/CD platform for AEM as a Cloud Service, to allow Apache/Dispatcher configurations to be deployed separately as their own pipeline – decoupled from deployment of the rest of the codebase.

There are a few reasons why this is an important feature, and a big deal.

Apache/Dispatcher “web tier” configuration updates typically happen much more often than full code releases.

The “web tier” here refers to all of the Apache configurations, which include rewrite rules, vhost configs, caching configurations, security filters, IP-level restrictions, Vanity URL configurations, and the like. And while there are many cases where a caching configuration in the Dispatcher has to go live at the same time as a code release, many other times there’s just a simple rewrite rule that needs to be added for the business, and it has nothing to do with the current release schedule. Decoupling the software release schedule from these configurations gives AEM teams the flexibility they need to get rapid day-by-day changes done while they keep on with established software delivery cadences.

The Current “Full Stack” Cloud Manager Deployment Process Takes Forever

A stumbling block that has been one of the major issues that teams migrating to AEM as a Cloud Service have had to deal with is the painfully-slow deployment times in Cloud Manager. Adobe’s Cloud Manager CI/CD solution is the only way to get code and configuration into AEM Cloud, and although it makes for easy single-click deployments that involve very little in the way of configuration to set up, the low-config / all-inclusive nature of the deployment means it can take a VERY long time to get your code out.

On Development environments, code deployments can take as little as 20 minutes (best case scenario) for very lightweight deployments, and as long as a few hours for heavier deployments. That means that for some deployments, you may only have enough time for one or two dev deployments in a workday.

Decomposing our most recent deployment to our sandbox Cloud Service environment, here were the timings involved:

• Validation – <1 minute
• Build & Unit Testing: 1 minute (re-used previous artifacts)
• Code Scanning: 3 hours 48 minutes (note: this time includes an unspecified time not recorded in the build logs waiting for someone to acknowledge/bypass the quality gates)
• Build Images for deployment: 24 minutes
• Deploy to Dev: 25 minutes

Or another Dev Deployment where there was more code flux and artifacts couldn’t be re-used:

• Validation: <1 minute
• Build & Unit Testing: 14 minutes
• Code Scanning: 4 hours 40 minutes (note: this time includes an unspecified time not recorded in the build logs waiting for someone to acknowledge/bypass the quality gates)
• Build Images for deployment: 20 minutes
• Deploy to Dev: 13 minutes

As you can imagine, this is already a fairly unacceptable time frame (mostly when compared to a lightweight in-house CI/CD process that one could create on AEM 6.5), but it’s significantly worse when you’re not even affecting the codebase with your change – i.e. let’s just say you’re adding a simple rewrite rule in Apache, or telling JPEG files that they can have a longer cache lifetime. For those simple configuration changes, there’s now the web-tier only pipeline that does significantly speed things up for this class of change.

The Web-Tier (Dispatcher/Apache Configuration) Pipeline

There are a few restrictions and technical prerequisites to using the web-tier-only or dispatcher-only pipeline. These, from the Adobe Documentation:

• You must be on AEM version 2021.12.6151.20211217T120950Z or newer to leverage web-tier config pipelines.
• You must opt in to the flexible mode of the dispatcher tools to leverage web-tier config pipelines.
• A user must be logged with the Deployment Manager role in order to configure or run pipelines.
• At any time, there can only be one web tier config pipeline per environment.
• The user can not configure a web tier config pipeline when its corresponding full-stack pipeline is running.
• The web tier structure must adhere to the flexible mode structure, as defined in the document Dispatcher in the Cloud.

In addition, be aware of how the full stack pipeline will behave when introducing a web tier pipeline.

• If a web tier config pipeline has not been configured for an environment, the user can make a selection while configuring its corresponding full-stack pipeline to include or ignore the Dispatcher configuration during execution and deployment.
• Once a web tier config pipeline has been configured for an environment, its corresponding full-stack pipeline (if one exists) will ignore the dispatcher configuration during execution and deployment.

In practice, the web tier deployment is indeed significantly faster than a full-stack deployment, though still somewhat variable in speed and sometimes still significantly longer than one would like.

As you can see below, the “Non-Production Pipeline” that deploys the full stack to Dev takes generally 1 hour to deploy. Note: The outliers (4 hours+) are including time spent waiting for one to override the quality gates to get the build to Dev, so that’s human wait time not actual build time.

What it appears (from the build log) that the web-tier pipeline does is:

• Check out your dispatcher configs and see if anything has changed
• Make a zipfile of all of the /conf.d, /conf.dispatcher.d, and /opt-in configs
• Run a scan of code quality rules against the dispatcher
• Test & validate the dispatcher configs with a mock service (example of some of that below)

Dispatcher is ready
Running tests against dispatcher on 127.0.0.1:80
End stage: wait for dispatcher
Begin stage: integration-test
Running [go test -v]...
2022/02/24 19:15:35 Expecting dispatcher to run at: http://127.0.0.1:80
=== RUN   TestMock
=== RUN   TestMock/Call_Mock_directly
--- PASS: TestMock (0.00s)
    --- PASS: TestMock/Call_Mock_directly (0.00s)
=== RUN   TestSystemReadyAccess
--- PASS: TestSystemReadyAccess (0.00s)
=== RUN   TestCommerceAccess
--- PASS: TestCommerceAccess (0.00s)
=== RUN   TestCRXAccess
    main_test.go:188: Skipping CRXAccess test. The proxyPass directive is not activated for ENVIRONMENT_TYPE 'prod'
--- SKIP: TestCRXAccess (0.00s)
=== RUN   TestCacheInvalidation
--- PASS: TestCacheInvalidation (0.00s)
PASS
ok  	backend-mock	0.012s

(A number of other mock tests are performed, including testing cache flushing, testing Commerce pass-through, testing virtual hosts, etc)

• Once tests pass, deploy out to Dispatchers

While there’s still room for improvement here, this is a huge new feature that I’m sure teams will appreciate. What I’m curious about is: for any of you with an existing AEM as a Cloud Service deployment, how does this change your team’s operation? Have you been doing your own internal validation and testing before code gets to Cloud Manager to limit some of the iterative deployment you have to do once the code is pushed to Adobe?