I am trying to put down some of the steps that we need to follow in any data migration projects. I have consolidated these steps after working on few migration projects and going through the some articles and documents that we used during our migration project cycle.
As per the various researchers data 65% to 70 % migration projects are failures 🙁 🙁 . These failures mainly occur due some very small information gap which later becomes the major issue.So lets us try to discuss the step oriented approach to avoid these failures and make a successful data migration delivery.
1) Source System
The first step is to indentify and explore the source system.
The best approach is to identify the group of data, account information, customer information and details, order and product details and other description based on the target application model.
The source system may contain thousands of fields; some might be good data, duplicate data or data that we may not require for the target application.
The source data may not contain all the data that we require for the target application we have clearly understand that gap and analyze from where we can get the require set to data, in some cases we may have to pull this set of data from the other third party source system etc. Finally we have to consolidate the data from the multiple sources and create the complete correct set of record to full fill the requirements for the target application.
The end of this phase you should be able to identify the source data that will populate the target model. You will also indentify the various gaps in the data set and consolidate these gaps using different source systems. Finally you will have to broken the data into various categories/entities to make the things more manageable. You will also identify the correct flow of data, that is, to load master data initially followed my parent entities and then child entities. Thus can also identify the various categories/entities that can we worked out in parallel.
2) Data Quality
The next very important step is to identify the quality of the source data. The new system may fail due data inconsistency or data redundancy, duplicate data thus making data migration unsuccessful.
Here is a important term called data profiling.
Data profiling is analyzing the contents of all the columns in the tables of interest. The data profiling will help us to identify the defects at the table and column level. It will help us in evaluating the correctness of data and ensuring the data quality as per the requirements of the target application.
Data profiling evaluate the actual record level and meta data information for the record set. Many projects start without analyzing the quality of the data of the source system.
By including the data profiling in very early stages of the project life cycle will help us to reduce the risk in the project , will give confidence to the migration team and client , will reduce project overruns, delays and to give timely successful delivery.
Thus at the end of this phase you will able to evaluate the quality of the data and if the source data is fit for the business process in the target model. You will also have the good understanding of the various data gaps and what correctness measures to be taken to fulfill those data quality gaps. You will also have the good understanding of mapping of the data from the source to target data model. Finally you will be able to develop the high level data integration plan.
3) Migration Design
In this step you will define the detailed technical architecture and design of the data migration process.
You should also provide some guidelines of the testing teams to be in sink with your data migration process.
In this it is important build all the detailed documentation for the project and takes all the required approvals to carry on the various activities successfully and timely.
4) Migration Strategies
The data migration strategies are the most important phase for the successful delivery for the project. Here you will define the details mapping of the source system to the target system.
The details of the various data messaging routines with proper set of data should be documented correctly and routines should be developed with considering all those business requirements in the data messaging routines.
The most important area that need to be covered is the testing , it is should explain how the testing should go first at low level with testing all the data by breaking it up into small categories/entities and then integrated testing of the various modules.
Output of this phase is fully tested process that can be delivered.
After running comprehensive testing the time comes to run the migration. In the majority of case the source system will be shutdown when migration takes place. To minimize the impact it is likely to occur on weekends or on public holidays. The synchronization for the source and target data will be validated.
Once the data is migrated, at some point of time you need to decide when to move the new system and where it is appropriate to retire old system. The synchronization of the data will be validated and checked before the actual transition happens.
As part of the design the system retirement policy will be created to address the old system. You will also manage the ongoing improvements and monitor the data quality of the new system.
Also try to identify the reusable components which can reused as well as possible improvements that can applied in the next migration project.
The data migration projects need to be handled with patience and leaders should show the confidence in the team because in any migration project you will get some very challenging issues and have look for work arounds for the fixing the issues.
Please feel free discuss the queries if you have any. We will try to give the good response as per our knowledge. and do provide your feedback on this article. 🙂 :)Thanks