Talend features to support Continuous Integration / Continuous Delivery – A first step to DevOps
The popularity of DevOps within IT organizations is ever growing. It has become a pre-condition for any company competing in the race for continuous (digital) innovation. New functionality or a new product offering needs to be put in the market by yesterday, according requirements, of high quality, and well-integrated into the existing system of applications. Once available, the offering needs to be reliable and easily adaptable to changing requirements. These are not just requirements that challenge the product itself, but also the process delivering the system enhancements. The DevOps mindset, tools and practices tackle these challenges by promoting a culture of continuous collaboration by means of automating the repetitive and predictive tasks involved in running a stable and effective DTAP street.
Where agile methods like scrum are already widely in use, DevOps entails a widening of agile principles and require another change in culture. As described above, this also brings changes in the processes and new functionality in the tools that accommodate these processes. That change does not happen from one day to another, and the advice would be to start small. CI/CD principles can already be implemented with manual steps. Therefore, in this part of the CI/CD series, we will describe some of the functionality that Talend incorporated in its latest releases to do so.
Artifact Repository
With direct support for Nexus Artifact Repository, it is possible to store stable job versions and move them through the different stages of the DTAP street. Why is this important? When scheduling the job execution directly from your source code repository, you might end up with untested functionality being deployed. In source control, frequent changes occur during development work. In case the Talend Administration Center (TAC) decides to re-compile a job, you could up with untested functionality being deployed. Stable job versions in addition to strict environment separation prevents this from happening. In addition to that, the use of self-containing jobs provides the portability required for purposes like testing, quality assurance and user-acceptance testing.
Well-integrated Git functionality
|
Git, as alternative to SVN, is not a precondition for CI, but it provides several benefits that come along with the various workflows. For example, a developer can check out a (feature) branch, allowing work on an isolated piece of code that can be merged back into the main code base later. The merge will occur only after he initiated a merge request for another developer to inspect his work. That developer can then switch onto the other developers branch to evaluate the changes, and approve the merge request. This type of workflow promotes the work on small batches of work, collaboration and fast feedback. |
 |
|
In case a developer merges a code change that has been touched by another developer as well, let’s say a changed context parameter, a merge conflict occurs. For such situations, Talend incorporated a Conflicts Navigator. This navigator lists the conflicts between the jobs, allow one to view the jobs side by side, both visually as code based, and provides the opportunity to accept or discard the conflicting items. |
 |
|
A integrated commit history window allows one to view the various changes of the specific branch or tag a developer is working on and view their details. All these functionalities, except for the merge request, are accessible directly via the studio, so rendering the need for a developer to switch interfaces. Another major advantage of working with git, despite being mentioned at the end, is that the interaction with the shared code base will prove to be a lot faster due to the de-centralized storage of project source code. |
 |
Test cases
|
By means of testing one tries to catch issues early during or after development, while the steps undertaken to incorporate a change are still fresh in mind. This reduces remediation costs in later stages. For a developer to check the functioning of one or more components of a job, Talend offers the functionality to create test cases directly from the Studio. These test cases can be regarded as unit tests since they test the system at the most granular level, on component level.To test a component handling a data flow, one can simply right click the component(s) and create a test case from the context menu. Several scenarios can be included by defining various test instances for a set of components. Scenarios then differentiate by the input data to be handled and the definition of the expected outputs. The comparison of the actual data versus these expected outputs form our test assertions. A developer can run these test via the studio of course, but they can also be scheduled to run via the TAC. |
 |
These new functionalities tend to make a Talend development team integrate their work more frequently, with more confidence and enhance the collaboration and visibility of their work. In other parts of the series we shine some light on automating steps in the delivery process and how (integration) test execution can be automated.
Contact us for a free demo or more information
Interested in more information? Contact us now or have a look at our upcoming events to meet us in person!