Continuous Integration (CI) and Continuous Delivery (CD) are increasingly being implemented across various development streams, while most BI projects are still being approached via the traditional waterfall method. However, CI and CD also offer great advantages in a BI environment. Discover the possibilities for your BI development line!
How can you apply CI and CD in a BI environment? You will find the answer to this question on 28 and 29 March, during the Data Warehousing & Business Intelligence summit. cimt, together with its partner Talend, is one of the participants at this unique annual DW/BI congress in Amsterdam. We will be happy to tell you all about the latest developments in the field of CI and CD in a BI environment. So make a note of the date in your agenda!
The advantage of CI and CD in a BI environment
In this blog we will lift a corner of the veil. So what exactly is the big advantage of CI and CD in a BI environment? The agile working method enables a team to add value in small iterations and with managed risk. CI provides faster feedback and increased confidence in the delivered code. Especially through integrated code review within the team. Because of this increase in confidence, the implementation of CD enables faster and more frequent deployments. Both with and without manual intervention.
From Continuous Integration to Continuous Delivery
Whichever development method you choose, the most important thing is that requirements are formulated as clearly as possible. But by making changes in small chunks, merging these packages of code (Continuous Integration) and testing the changes automatically (Continuous Testing), changes are brought to production faster in a controlled and phased manner (Continuous Delivery). When this process is fully automated, we talk about Continuous Deployment.
Starting with CI and CD
What do you need to implement CI and CD in a BI environment? One indispensable tool is the Talend Open Studio. In addition, the Talend Administration Console and generic CI/CD components such as a software version control system (SVN/GIT), Nexus and a CI server like Jenkins are also indispensable.
Requirements for CI and CD
To achieve CI and CD, the right facilities need to be in place, of course. But also the right mindset and the new “way of working” must be adopted, so that the transition to DevOps can be initiated.
How does it work in practice?
Developers building ETL jobs, for example, are constantly working to improve these jobs or enrich them with new functionality. All jobs built in Talend are translated into code. This code can easily be stored in GIT. In addition to integration with SVN, Talend offers an integrated solution for working with GIT. GIT offers more freedom to design your own workflow and faster work through a local copy of the repository than SVN.
If the developed code passes the unit test, the “build” can be stored in Nexus as an artifact. These artifacts can then be deployed in the next “stage” (develop, test, accept, produce).
Direct feedback
Deploying the artifact in an environment with more variables (e.g., acceptance) reveals whether the developed work functions properly in this environment. Should the automated test fail, the development team receives immediate feedback. An important role is reserved for the CI server (e.g. Jenkins). The CI server should be seen as the conductor of the delivery team. It starts, stops and gives feedback on processes. By automating all processes, the chance of errors is reduced, the feedback loop shortened and the speed increased.
Curious how we implement this with clients?
Visit us at the Data Warehousing & Business Intelligence summit and our client will explain it for you!