Transitioning to a Big Data environment
Organizations are producing and storing more and more data for analysis now or at a later date. However, traditional relational database management systems, which many companies still use, are poor at handling large amounts of semi-structured and unstructured data such as log data, sensor data and social media (including Google Adwords and Analytics, Twitter and Facebook). New ecosystems of solutions from the Open Source Community, known by the collective names “Hadoop” and “NoSQL,” as well as so-called “Big Data” platforms, offer perspective.
Government agencies, financial service providers and companies in the (tele)communications sector in particular are already dealing with the processing of large amounts of different data types. Traditional databases, however, offer limited scalability and storage options for the explosively growing mountain of semi-structured and unstructured data. For such organizations, switching to a ‘NoSQL’ database can therefore be the solution. In addition, “Big Data” platforms have now emerged that can handle both the old and the new situation well. Thus, these platforms can facilitate a transition of data to the new ecosystems, among other things.
Benefits of Big Data Platforms
- They focus on processing and storing large amounts of unstructured and semi-structured data and complex data types.
- These platforms enable high speed data storage and processing.
- Solutions are easily available because they are developed from an “open source” model.
- There is a great deal of focus on developing and supporting the new ecosystem, which will allow both these systems and the Big Data platforms to mature rapidly.
Switching to a Big Data environment?
When transitioning to a Big Data platform, it is important to consider a number of things:
- The connectivity between the applications and the NoSQL database.
- Modeling the data and data structures.
- Knowledge in the organization regarding operational management of NoSQL databases and/or Hadoop, among others.
- Migration from the current data platform to the new Big Data platform.
A successful start
The first step for a successful transition is to set up a pilot environment. This will allow users to become familiar with the new Big Data platform. If, after the pilot phase, the final decision is made to migrate to the new platform, it must be determined how the data can be migrated from the relational database and what strategy is appropriate.
Transitioning to a Big Data environment: our strategy
With years of experience in the field of data integration, cimt can help organizations transition to a Big Data platform. Here, a step-by-step approach can be a good strategy. Data from one application or database is migrated to the Big Data environment. Starting small and taking it one step further each time, for an optimal result in a relatively short time frame.
Want to read more about transitioning to a Big Data environment? Then download our white paper.