Planning for Successful Data Migration

In previous chapters, I’ve commented “This is the most important…” when it came to areas that can cause issues during a project, and I’m going to repeat that. Data is what makes the system run. Without data, the system is unusable.

When building a new system, much discussion, planning, and testing needs to be completed to make sure data is imported into the new system correctly. In this chapter, we’ll discuss the major impacts and pitfalls related to data migration and data translation when moving into D365. We’ll also look at tools and potential issues that can arise when moving data.

The main topics to learn about in this chapter are as follows:

• What is data migration?
• Data migration methodologies
• Data management
• Creating a migration strategy
• Data migration best practices
• Case studies

What is data migration?

One question that I get asked a lot from inexperienced solution architects is “Why do I care what the data looks like? It’s not my problem as a solution architect.” Well, that’s not exactly true. As we’ve mentioned in a previous chapter, the solution architect is responsible for everything in the project, and data management/migration is one of the most important. I guess first off, we need to understand what data migration is. The simplest way to look at it is that data migration is the process of moving data from an existing system and importing it into D365.

And that data can come in many forms. It could be in an existing system, Excel spreadsheets, or flat files. Or even better, it might exist on paper in file cabinets (those projects are the ones where I want to cry). In any case, the data needs to be put into D365. But not all data. The only type of data that should be migrated is what is called master data. We’...

Data migration methodologies

Let’s use an example of a company moving from a simple desktop accounting program to D365. All the data that they have in the existing software is only related to financial transactions. The version of their accounting software uses a localized database for all of the records and does not have a way to connect to external systems. All of the inventory they have in their warehouse is kept in a number of Excel spreadsheets. They also use the FedEx© online tool for calculating shipping costs when they have to ship items out to customers. They find that they are often retyping information into the online tool and are very inefficient.

The spreadsheets are often incorrect as well, due to the manual nature of how they have to be updated. Often, two or more people will be picking items for shipping. Once the first picker takes an item and the second goes to pick something, they will often find the inventory list doesn’t match what’...

Data management

To assist in the migration of data from different environments, we have several different toolsets at our disposal that make the process much easier. These toolsets can be broken down into three different types:

• The data management framework: This is a set of tools that are located on the main screen of the D365 F&SCM home page. This workspace contains all the functions needed to work with data packages.

Figure 5.2: Data management workspace

• Data sharing functions: These tools are used to help migrate and synchronize data between different legal entities in the same company.
• Standard database operations: In this toolset, we use the standard database functions of backup/restore. We may also use these tools to do a data rollback, or a point-in-time recovery.

Figure 5.3: Data toolsets

Data management framework

The data management framework is a set of tools and procedures that are...

Creating a migration strategy

Now that we’ve looked at a few of the needs and tools for data management, let’s change focus and look at how to create a data migration strategy that will be successfully executed during a project. We’ve also identified where the data lives, and what tools we plan on using. We need to break the migration strategy into three phases:

1. First, define what the migration strategy will be.
2. Second, create a cutover plan for go-live.
3. Third, determine how much and what data is to be kept as part of a data retention plan.

Prepare for data migration

As we get closer to the point where we can start to migrate data, having a detailed plan will simplify the life of both the SA and the client. There are several tasks that need to be completed so that we can accomplish both the migration of the data and also the client’s business continuity.

Note

The processes involved in data cleanup and extraction are long....

Data migration best practices

We’ve talked a lot about some of the features of the data migration and management parts of the project. There are lots that we haven’t talked about as well. But one thing that we need to discuss is Microsoft’s best practices for data management. There are several of them, and we’ll discuss each one, including my comments and recommendations for each one. These recommendations have been accumulated from my and fellow SAs’ projects.

First and foremost, never ever use the production environment to test data migration. The data should be put into a backup system or database, if possible, and use that system to the data extraction parts of the migration tasks. Also, a Tier 2 system should be used for the majority of the development of the data packages. Once you’ve tested the data against backup and test environments and you’ve validated everything, when go-live comes, you can use the data packages against...

Case studies

Before we end this chapter, let’s look at a couple of projects I was involved in that were good and bad respectively. The good one worked well due to amazing planning and cooperation from the client. The second one didn’t work out quite that way and was a major reason why the project went on much longer than planned.

Let’s look at the successful one first. One of the first projects I worked on as an SA was with a client who had a small finance system in place and a manual system for everything else. Let’s give a bit of background.

The company whose project I worked on was an international airport. After extensive searching, they decided on going with Dynamics as their ERP system. The company had approximately 300 employees who would be using the system. As was already mentioned, their finance system was an existing system that had been in place for several years. All their inventory and purchasing were completed in manual systems that...

Conclusion

Hopefully, after reading this chapter, you’ll see how important data migration and management are to a project. If you don’t look at data starting right from the beginning, it could very likely come back to delay you later on in the project.

In the next chapter, we’ll take a good look at some of the licensing challenges that can occur in your project.