“It’s a production application in development,” explained a development manager to a production control administrator, “and that is why we need to do our software releases so frequently.” I watched as the production control team looked at the development manager like he was crazy.
A production application in development can be a difficult concept for a production control administrator to fully grasp or support. For development, a production application should be fully baked and not in what would be considered a “development” state. However, frequent releases are a basic requirement of rapid development methodologies like agile and this impacts the way in which development teams and production control teams must interact.
In essence, applications go to production before they are fully baked, and they get better over time until the last iteration has been completed at which time the application goes into a “maintenance mode” where scheduled releases are approved by the production control team, but often managed by developers. Production control teams are ready to support scheduled releases, but struggle for many reasons with continuous deployments.
For example, production control prefers to manage releases in a stricter manner, including an approval process that may take one to two days. The application moving to production is added to a release calendar and scrutinized through release meetings and discussions. It is not unusual for the production control team to hold a release, just to make sure new changes are not added. This is contrary to continuous deployment where frequent small releases are moved to production in a more automated fashion with less scrutiny or discussion.
Production control teams who support distributed platforms have never fully taken over the task of performing the build to deploy steps for managing production binaries. In many large companies the production control team serves as an “approving” body, providing the development team access to production servers for the purpose of managing its releases. When releases were less frequent, this potential violation of the separation of duties flew “under the radar.”
If there was a problem with production, the production control team was responsible for fixing the issue and answering to upper management, always with the development teams doing the work to get production back on track.
With the introduction of agile development, where frequent releases are a core part of the process, the production control landscape has changed. Production releases are more likely to cause upper management to question the frequency of releases to the production environment. The answer to the question, “Why frequent releases?” is “It’s a production application in development.” Yes we have come full circle, which is why the DevOps conversation has become much more prominent in discussion blogs, technical journals, and in the conference rooms of companies looking to improve the moving of applications between development and production control.
Frequent releases are being reported to upper management exposing that there is a change in the way applications are moved into the production environments. Instead of scheduled releases, production control is being asked to push through frequent releases changing the standard release processes that production control has followed for the last thirty years. While agile teams argue that smaller, more frequent releases reduce risk, upper management may see frequent releases as an unstable, high-risk production environment.
At its core, DevOps attempts to streamline and simplify the hand-off of the software application between development and production as well as calm the nerves of upper management who are wary of frequent releases. Giving production control what they need to manage a production application in development is critical to simplifying the hand-off. Of the most critical information that production control needs from development is the ability to confirm that the application has been built on the correct production level of the underlying technology stack.
They want to see that the executables have matching source code and can be rebuilt for an emergency release or a corporate audit. They often want sign off from a testing team that confirms that the new release has been fully tested in an environment that models production. Details on server configurations, environment variables, and release steps are needed from the development teams for production control to repeat the deploy process at the production state.
To satisfy upper management, production control needs to report and manage key facts about the applications moving into production. Production control members need to report that the new release is compatible with the production environment, that emergency fixes are possible, and that there is an audit trail for tracking who requested the release, the time and date of the release and what new features or fixes initiated the release often down to what source code was updated and what new binaries were moved to production for the new release. Production control needs to confirm that the application moving to production is built on the production technology stack, not the development technology stack.
Additionally, production control members need a clear topography of the servers that are used, the steps necessary to perform the deployment, and details about the server’s configuration. Even though the team members may not themselves execute the deployment, they are responsible for it. This means that team members need the knowledge of what is occurring at the production deployment level. In other words, production control needs basic information about how applications are built and deployed.
A lack of shared information is the primary reason why the hand-off between development and production is problematic. This information is hidden in one-off script-driven processes that manage both the build and deploy. These one-off script driven processes are known by only a few individuals on the development team, and in most cases cannot be known to production control. Furthermore, these processes impede the sharing of control between development and production. Ultimately, this creates some nasty politics between the two teams. What appears to be a constant struggle for power, in reality is simply a struggle for shared knowledge and control.
Most organizations use scripting to manage builds and deploys. Each development team determines how their build and deploy scripts will operate. Because of this, a company with fifty java teams can have fifty plus build and deploy scripts that look and act completely different. All scripts can be building and deploying a Jboss application, but they do it in their own unique way. And no, there is never a reason for the difference, other than the names of files and servers. The Java compiler works the same regardless of the file name. JBoss applications are deployed the same way every time with the exception of binary names and some server configurations. Each developer who writes his application’s build and deploy scripts, simply manages the logic in different ways making the scripts non-standard and often hard to debug, fix, and re-execute.
Some developers do an outstanding job of writing their build and deploy scripts, and others not so much. As a result, production control must manage a web of scripts with no standards, no common reports, no method for matching source to binaries, no method for validating the technology stack, or no method for repeating a build or deploy if needed. Thus, the constant struggle for control of the deploy process continues between teams. Production control team members asks for what they consider to be basic information, which development teams are unable to produce in a standard and repeatable way across the organization. Static scripts hide critical information such as the technology stack dependencies and generate no common reports that should be shared across team.
We must strive towards using real automation and standardization. A one-off script executed by a job-scheduler or workflow tool is not real automation. Creating re-useable build and deploy scripts will ultimately be the answer for delivering standardization to both build and deploy. Today’s DevOps solutions are quickly beginning to address the problem of one-off scripts and lack of repeatability. There are DevOps solutions that address both the build and deploy by generating your build and deploy scripts in a consistent, repeatable, and transparent way.
These methods deliver standard reports, continuous build and deploy, incremental processing, accelerated builds, parallelized deploys, the management of server configuration, and the validation of the correct technology stack as part of the build and deploy process. In essence, the tools that move away from one-off scripts are providing a framework for sharing the build and deploy details between development and production so that the hand-off is indeed simplified through shared knowledge and control.
Moving away from one-off script driven processes is not entirely new. We simply need to look at the transformation the mainframe teams went through in the early 1980s. Compile and deploy Job Control Language (JCL) used to be the most common way for build and deploys to occur on the mainframe. It is now extremely uncommon to see private compile and deploy JCL used on the mainframe. You may argue “the mainframe is easier to manage so they can standardize their scripts,” but as a person who has done both, I would disagree.
There are as many variables on the mainframe as there are in the distributed environments from Windows to Unix to Linux. And yes, mainframers can do a check-in and build (continuous integration) and move only the updated files through the lifecycle quickly (incremental continuous deploy) while production control manages the entire process without using a single one-off JCL script. Ultimately, a similar process will be adopted on the distributed platforms allowing production control teams and development teams to share the knowledge and management of software builds and deploys.
Developers can then spend less time and money on writing one-off script driven processes and instead focus on what they do best—quickly deliver awesome business software to maximize the profits of their employers.