Chapter 6. Data Management

In this chapter, we will cover the following recipes:

• Data entities
• Building a data entity with multiple data sources
• Data packages
• Data migration
• Import of data
• Troubleshooting

Introduction

The data management feature in Dynamics 365 for Finance and Operations enables you to manage and audit your data efficiently in systems. The excellent feature provides many tools such as Import, Export, delete bulk data and detect duplicate data, and so on. You can also develop custom data entities as well.

Integration through the data management platform provides more capabilities and higher throughput for inserting/extracting data through entities. Typically, data goes through three phases in this integration scenario:

• Source - These are inbound data files or messages in the queue. Typical data formats include CSV, XML, and tab-delimited.
• Staging - These are automatically generated tables that map very closely to the data entity. When data management enabled is true, staging tables are generated to provide intermediary storage. This enables the framework to do high-volume file parsing, transformation, and some validations.
• Target - This is the data entity where data will be imported...

Data entities

In the earlier version of Dynamics 365 for Finance and Operations, there are multiple options such as DIXF, Excel Add-ins, and AIF for data management. Data entities are introduced as a part of data management to be used as a layer of abstraction to easily understand by using business concepts.

The concept of data entities combines those different concepts into one. You can reuse Data entities for an Excel Add-ins, Integration, or import/export. The following table shows core scenarios of Data management:

Data Migration	Migrate reference, master, and document data from legacy or external systems.
Setup and copy configuration	Copy configuration between company/environments. Configure processes or modules using the Lifecycle Services (LCS) environment.
Integration	Real-time service based integration. Asynchronous integration.

Note

More information about this can be found at https://docs.microsoft.com/en-us/dynamics365/unified-operations/dev-itpro/data-entities/data-entities-data-packages...

Building a data entity with multiple data sources

We could also create a data entity where we include multiple data sources. Here our data entity takes care of all integrity constraints and validation and creates records in related tables if it does not exist. Let us take, an example of inventory breakdown, where we create an inventory site, warehouse, location, zones, aisle, and so on. We could create a data entity, which encapsulates all these tables, and a flat file import could create related records in all these tables.

How to do it...

Carry out the following steps in order to complete this recipe:

1. Add a new data entity in the project and name it PacktInventBreakDown.

2. A Data Entity Wizard will be launched, as shown in the following screenshot:

3. Next you need to select all/required fields from WMSLocation.

4. Click on the Add data source button and select Relation InventLocation.

5. Select Invent location from the node at the right of WMSlocation, as shown in the next screenshot.
6. Add new data source...

Data packages

Data packages in Dynamics consist of logically grouped data entities. In a simpler way, a data package contains one or more entities or groups of data entities. Lifecycle Services (LCS) contains multiple base data packages that you can use to reduce implementation time during the project. These packages also be used to prepare your system in much less time with demo/real data. These packages can contain the elements that are required in each module/area in order to meet the minimum requirements. As per business requirements or advanced business processes, you might have to add more entities to the list of packages.

Getting ready...

The data packages that Microsoft publishes on LCS use a numbering sequence that is based on the module, data type, and sequence. Here is an example:

• Module/area numbering:

• Data type numbering:

• Numbering format:

The names of data packages include the numbering format, which is followed by the module abbreviation and then a description. For example, the...

Data migration

Data migration is a key task of a project implementation cycle. Here are the key points regarding export/import tasks.

The following pain points can occur during migration:

• Inability to do quick iterative migration and validations
• Multiple hops that lead to multiple dependencies and change of errors
• Complexity due to repeated manual interventions
• Difficulty in tracing and error troubleshooting
• Difficulty migrating a large volume of data within a time constraint

During migration, you can strategize and choose data entities. Data entities also save time during implementation because previous activities required data export from a database, data export validation, and data transformation to files such as Excel or XML. In the current version of Dynamics 365 for Finance and Operations, these hops have been eliminated. If an import error occurs, you can skip selected records and choose to proceed with the import using the good data, opting to then fix and import the bad data later. You...

Importing data

Import is the process of pulling data into a system utilizing data entities. The import process is done through the Import tile in the Data Management workspace. Data can be imported either for individual entities or for a group of logically related entities that are sequenced in the correct order. The file formats vary depending on the type of import. For an entity, it can be an Excel file that is comma-separated, tab-separated, text. For a data package, it is a ZIP file. In both cases, the files are exported using the previously-mentioned export process. The detailed steps for importing data using data packages are as follows.

How to do it...

Let us import and create released products in Dynamics 365 for Finance and Operations:

1. In the system administration module, click Data management workspace, to begin importing, select import tile.
2. In the Name field, provide a logical name for the package, which is being imported. In the Source Data Format field, select CSV as the source...

Troubleshooting

Now you must have understood the concept and logic behind data management. We also created a few entities, packages, and other related objects. Import and Export become very easy with Dynamics 365; however, these processes are not always straightforward. You may get a few unwanted errors during this.

In this section, we will see how to troubleshoot these errors and warnings.

Getting ready

While working with data management at times we encounter errors in import and exports. The framework provided with the product is robust enough to identify the error logs and suggest for possible solutions to rectify the data. This section describes how to troubleshoot during the different stages of data package processing.

How to do it...

Carry on the following steps to troubleshoot some issues that you may get during routine development:

1. Normally during export processes, we do not get any errors, but if you get an error during the export process click View execution log and review the log text...

The rest of the chapter is locked

Building a data entity with multiple data sources

We could also create a data entity where we include multiple data sources. Here our data entity takes care of all integrity constraints and validation and creates records in related tables if it does not exist. Let us take, an example of inventory breakdown, where we create an inventory site, warehouse, location, zones, aisle, and so on. We could create a data entity, which encapsulates all these tables, and a flat file import could create related records in all these tables.

How to do it...

Carry out the following steps in order to complete this recipe:

1. Add a new data entity in the project and name it PacktInventBreakDown.

2. A Data Entity Wizard will be launched, as shown in the following screenshot:

3. Next you need to select all/required fields from WMSLocation.

4. Click on the Add data source button and select Relation InventLocation.

5. Select Invent location from the node at the right of WMSlocation, as shown in the next screenshot.
6. Add new data source...

Data packages

Data packages in Dynamics consist of logically grouped data entities. In a simpler way, a data package contains one or more entities or groups of data entities. Lifecycle Services (LCS) contains multiple base data packages that you can use to reduce implementation time during the project. These packages also be used to prepare your system in much less time with demo/real data. These packages can contain the elements that are required in each module/area in order to meet the minimum requirements. As per business requirements or advanced business processes, you might have to add more entities to the list of packages.

Getting ready...

The data packages that Microsoft publishes on LCS use a numbering sequence that is based on the module, data type, and sequence. Here is an example:

• Module/area numbering:

• Data type numbering:

• Numbering format:

The names of data packages include the numbering format, which is followed by the module abbreviation and then a description. For example, the...

Data migration

Data migration is a key task of a project implementation cycle. Here are the key points regarding export/import tasks.

The following pain points can occur during migration:

• Inability to do quick iterative migration and validations
• Multiple hops that lead to multiple dependencies and change of errors
• Complexity due to repeated manual interventions
• Difficulty in tracing and error troubleshooting
• Difficulty migrating a large volume of data within a time constraint

During migration, you can strategize and choose data entities. Data entities also save time during implementation because previous activities required data export from a database, data export validation, and data transformation to files such as Excel or XML. In the current version of Dynamics 365 for Finance and Operations, these hops have been eliminated. If an import error occurs, you can skip selected records and choose to proceed with the import using the good data, opting to then fix and import the bad data later. You...

Importing data

Import is the process of pulling data into a system utilizing data entities. The import process is done through the Import tile in the Data Management workspace. Data can be imported either for individual entities or for a group of logically related entities that are sequenced in the correct order. The file formats vary depending on the type of import. For an entity, it can be an Excel file that is comma-separated, tab-separated, text. For a data package, it is a ZIP file. In both cases, the files are exported using the previously-mentioned export process. The detailed steps for importing data using data packages are as follows.

How to do it...

Let us import and create released products in Dynamics 365 for Finance and Operations:

1. In the system administration module, click Data management workspace, to begin importing, select import tile.
2. In the Name field, provide a logical name for the package, which is being imported. In the Source Data Format field, select CSV as the source...

Troubleshooting

Now you must have understood the concept and logic behind data management. We also created a few entities, packages, and other related objects. Import and Export become very easy with Dynamics 365; however, these processes are not always straightforward. You may get a few unwanted errors during this.

In this section, we will see how to troubleshoot these errors and warnings.

Getting ready

While working with data management at times we encounter errors in import and exports. The framework provided with the product is robust enough to identify the error logs and suggest for possible solutions to rectify the data. This section describes how to troubleshoot during the different stages of data package processing.

How to do it...

Carry on the following steps to troubleshoot some issues that you may get during routine development:

1. Normally during export processes, we do not get any errors, but if you get an error during the export process click View execution log and review the log text...