1. Data Management
  2. Data Quality Studio

Activities

Activity Area Description

Activate data quality policy version

 		Data quality

When the full setup of a data quality policy version is finished, to apply the defined rules, you must make the version active.

Note: You cannot edit an active data quality policy version.

Check for, review, and merge duplicates - Fuzzy matching

 		Data quality

A Fuzzy matching duplicate check checks for duplicates applying fuzzy logic. A Fuzzy matching duplicate check compares, for a selected record, several field values with the values of the same fields of other records.

When duplicate values are found in another record:

  1. The duplicate score is calculated. The duplicate score is calculated based on the field weightage as defined for the duplicate check.
  2. The calculated duplicate score is compared with the threshold as defined for the duplicate check.
  3. If the duplicate check is equal to or higher than the threshold, the record is reported as possible duplicate.

Example:

Duplicate check on CustTable

Threshold: 50%

Table name Datasource name Field Field label Weightage
CustTable CustTable AccountNum Customer account  
CustCustomerV3Entity CustCustomerV3Entity AddressStreet Street 1
CustCustomerV3Entity CustCustomerV3Entity AddressZipCode ZIP/postal code 1
CustCustomerV3Entity CustCustomerV3Entity OrganizationName Organization name 6
CustCustomerV3Entity CustCustomerV3Entity PrimaryContactEmail Primary email 3
CustCustomerV3Entity CustCustomerV3Entity PrimaryContactPhone Primary phone 3

Calculation examples:

  • Duplicate values exist in the Primary email field and in the Primary phone field. The duplicate score is: 6 / 14 * 100 = 42,86. The record is not reported as possible duplicate.
  • Duplicate values exist in the Organization name field and the ZIP/postal code field. The duplicate score is: 7 / 14 * 100 = 50. The record is reported as possible duplicate.

Review and merge duplicates

You can review the found duplicates.

To solve duplicates, you can:

  • Merge field values from the duplicate records to a chosen master record.
  • Manually delete undesired duplicate records.

Clean up fuzzy duplicate check history

 		Data quality

If you have checked a record for fuzzy duplicates, you can view the fuzzy duplicate check history for that record. The history shows all fuzzy duplicate checks that are done for the record.

You can clean up the logged fuzzy duplicate check history manually or in recurring mode.

For example, you want to keep fuzzy duplicate check history records for one month. Each week, you can do a cleanup, deleting history records older than one month.

The steps of this topic explain how to clean up the fuzzy duplicate check history.

Clean up validation history

 		Data quality

In the Data quality studio parameters, you can enable logging of data quality policy validation rules execution.

Validation rule execution results are logged only when you manually add or change records. Each time a validation rule is executed, an entry is logged.

You can clean up the data quality policy execution log once or in recurring mode.

For example, you want to keep validation history records for one month. Each week, you can do a cleanup, deleting history records older than one month.

The steps of this topic explain how to clean up the validation history.

Copy data quality policy

 		Data quality

Use data quality policies to ensure that your data is of the desired quality. You can organize data quality policies, for example, by functional area, business process owner, or organization assignment.

To create a data quality policy, you can copy an existing one. When you copy a data quality policy, you must choose which versions you want to copy. In the new data quality policy, the chosen versions are added in the version sequence of the original data quality policy. Example: You copy versions 4 and 6 of the original data quality policy. In the new data quality policy, version 4 is added as version 1, and version 6 is added as version 2.

Create data quality policy

 		Data quality
Use data quality policies to ensure that your data is of the desired quality. You can organize data quality policies, for example, by functional area, business process owner, or organization assignment.
If you create a data quality policy, its header and first version are created.

Create data quality policy version

 		Data quality
You can create a new version for a data quality policy. If you create a new version, you can choose to:
  • Use an existing version as basis. As a result, all elements of the selected version are copied to the new version.
  • Create a blank version.
Note: If you create a data quality policy, automatically its first version is created.
A data quality policy version contains all elements of a data quality policy, except for the data quality policy header.
These elements are version-specific:
  • Validation rules
  • Duplicate check rules
  • Action rules
  • Organization association

Create dynamic query with the Wizard

 		Data quality
Use a dynamic query to find one or more records in the database. The found records are the input for further processing.

You can use a dynamic query on:

  • An action rule of type Data query.
  • A condition that is applied to a validation rule, duplicate check rule, or action rule.
This topic explains how to set up a dynamic query with the Query wizard.
You can also start the dynamic query wizard from an existing query to edit it.

Create enrichment rule

 		Data quality

Use enrichment rules to set field values in D365 F&SCM.

This topic explains how to create an enrichment rule.

Define to which forms fuzzy duplicate check rules apply

 		Data quality

You can start a duplicate check rule of type 'Fuzzy matching' only manually.

To indicate the forms from which you can start a duplicate check rule of type 'Fuzzy matching', set up the dynamic menu form setup.

To each form in the dynamic menu form setup, these buttons are added:

  • Check for duplicates: Starts the first found active Fuzzy matching duplicate check that:
    • Applies to the main table of the form.
    • Is used in a duplicate check rule of an active data quality policy.
  • History: Opens the duplicate check history of the selected record.

Button style

In the dynamic form setup, use the Style to define where and how the 'Check for duplicates' button and the 'History' button are shown on the form:

  • New action pane tab: To the Action Pane, the 'Data quality' tab is added as the first tab. This tab shows the 'Duplicate check' button group with the buttons.
  • New button group: The buttons are shown on the task bar, right to the existing buttons. 
  • Existing action pane tab: To an existing Action Pane tab, the 'Duplicate check' button group is added. This button group shows the buttons. In this case, you must select the Action Pane tab to which the 'Duplicate check' button group must be added. You can indicate next to which existing button group the 'Duplicate check' button group is added. If you don't do so, the 'Duplicate check' button group is added as the first button group of the Action Pane tab.

Prerequisites

On a form with dynamic menu form setup, you can start a duplicate check rule of type 'Fuzzy matching' only if:

  • The duplicate check rule is part of an active data quality policy.
  • The duplicate check rule uses an active duplicate check of type 'Fuzzy matching' that applies to the main table of the form. The table to which the duplicate check applies is defined in the duplicate check header, in the Table name field.

Define web service setup for enrichment rule

 		Data quality

Use a web service response enrichment rule line to get a field value from another internal or external source, using a web service, and set this value in the target field of the enrichment rule line.

Before you can set up a web service response enrichment rule line:

  • Set up the web service to be used.
  • Define the web service that applies to the enrichment rule. Also define the desired request parameters setup. The enrichment rule web service and related parameters setup is explained in this topic.

Request parameters

For a web service, request parameters can be set up. If so, these request parameters are shown on the Web service setup page, in the Request parameters section. Note: If a Custom request handler is defined for the web service, no Request parameters are available.

You can set up and apply the request parameters in several ways:

  • Use the default parameter values, as defined for the applicable web service. In this case, no specific web service parameter setup is required for the enrichment rule.
  • Override a default parameter value. To do so, you can use one of these Parameter types: Field, Custom, or Fixed value. Each type requires specific setup.

 

Delete data quality policy

 		Data quality

You can delete or retire a data quality policy.

If you try to delete a data quality policy:
  • With an active version, it is neither deleted nor retired.
  • With only draft versions, and at least one of these versions was active in the past, it is retired instead of deleted. To view the retired data quality policies, on the Data quality management workspace, click the Retired data quality policies tile.
  • With only draft versions, and none of these versions were active in the past, it is deleted.

Develop custom action class

 		Data quality You can use a custom action rule to set a field value using a custom class.
To set up a custom action rule, first develop the custom class to be applied.
The custom class must extend the DQSCustomValidationBase base class.
When you extend the DQSCustomValidationBase base class:
  • Enter a unique and recognizable description() for the class. The description is shown in the lookup of the Class name field of a custom action rule.
  • Use these methods to indicate to which data the custom class is applicable:
    • getTablesList(): You can define the tables to which the custom class applies. As a result, you can select the custom class for a custom action rule with one of the applicable tables defined.
    • getExtendedDataTypesList(): You can assign extended data types to which the custom class applies, for example, 'ExternalItemId'. As a result, you can select the custom class for a custom action rule with a target field defined that uses the same extended data type.
    • getBaseTypesList(): You can define base data types to which the custom class applies, for example, String or Date. As a result, you can select the custom class for a custom action rule with a target field defined with the same data type.
  • Use the processRecord() method to define the x++ code that defines the return value that is used as input value for the target field of the action rule.
The base structure of an extended DQSCustomValidationBase class is:

Develop custom parameter value class

 		Data quality To define a web service parameter value, you can use a custom class to return a parameter value.
You can use a custom parameter value class to return web service parameters for validation rules and action rules of type Web service. 
The custom parameter value class must extend the DQSWebServiceParametersBase base class. This custom class must have at least one return method that returns a value.

Develop custom response or request handler class

 		Data quality

For a web service, you can use a custom class to define the:

  • Request that you send to the web service. You can, for example, use a custom class to add some binary code or other logic to the web service request. Use the PostData method to define web service requests.
  • Response that you expect from the web service. You can, for example, use a custom class to add some binary code or other logic to the web service response. Use the parseResponse method to define web service response.
The custom class must extend the DQSWebServiceCustomFormatBase base class.

Develop custom validation class

 		Data quality
You can use a custom validation rule to validate a field value using a custom validation class.
To set up a custom validation rule, first develop the custom validation class to be applied.
The custom validation class must extend the DQSCustomValidationBase base class.
When you extend the DQSCustomValidationBase base class:
  • Enter a unique and recognizable description() for the class. The description is shown in the lookup of the Class name field of a custom validation rule.
  • Use these methods to indicate to which data the custom class is applicable:
    • getTablesList(): You can define the tables to which the custom class applies. As a result, you can select the custom class for a custom validation rule with one of the applicable tables defined.
    • getExtendedDataTypesList(): You can assign extended data types to which the custom class applies, for example, 'ExternalItemId'. As a result, you can select the custom class for a custom validation rule with a field defined that uses the same extended data type.
    • getBaseTypesList(): You can define base data types to which the custom class applies, for example, String or Date. As a result, you can select the custom class for a custom validation rule with a field defined with the same data type.
  • Use the validateRecord() method to define the x++ code that runs the validation. This code must return one of these values:
    • True: The value in the validation rule field is valid.
    • False: The value in the validation rule field is invalid.
The base structure of an extended DQSCustomValidationBase class is:

Develop function class

 		Data quality You can use a function to fill the value of a condition on a query range.
For a function, a class is required. To develop this class, extend the DQFFunctionRun class.
Prerequisite: Develop the extension of the DQFFunctionRun class in the AOT before you start the steps.
Note: The steps are to have the developed function class uploaded as function.

Export data quality policy version

 		Data quality

You can export a data quality policy version to an XML file. So, you can import it in another Microsoft Dynamics 365 for Finance and Supply Chain Management environment.

If you export a data quality policy version, all related settings, like rules, organization assignments, and conditions, are included.

Finish quality assessment result review

 		Data quality

As a data owner, when you have finished the review and editing of a data quality assessment record with errors or warnings, change its status accordingly.

You can choose one of these statuses:

  • Accept deviation: You did not make changes to the source record, nor merged duplicate record fields to solve the error or warning. So, you accept the deviation from the data quality policy rule.
  • Completed: You made changes to the source record or merged duplicate record fields to solve the error or warning.

Import data quality policy version

 		Data quality

You can import a data quality policy version from an XML file. This XML file must be the result of a data quality policy version export on another or the current Microsoft Dynamics 365 for Finance and Supply Chain Management environment.

The XML file contains the exported data quality policy version, with all related settings, like rules, organization assignments, and conditions.

If, in the environment, the imported data quality policy:

  • Exists, the imported version is added to this data quality policy as a new version.
  • Exists as a retired data quality policy, you cannot import the data quality policy. However, you get a message with the question if you want to restore the retired data quality policy. If you click OK, the retired data quality policy is restored. Also, the imported version is added to the restored data quality policy as a new version.
  • Does not exist, the imported data quality policy is created with a first version.

Monitor fuzzy duplicate check history

 		Data quality

If you have checked a record for fuzzy duplicates, you can view the fuzzy duplicate check history for that record. The history shows all fuzzy duplicate checks that are done for the record.

If values are merged from duplicate records to the selected record, you can view the merged values.

Monitor validation history

 		Data quality

In the Data quality studio parameters, you can enable logging of data quality policy validation rules execution.

Validation rule execution results are

  • Logged only when you manually add or change records. Each time a validation rule is executed, an entry is logged.
  • Not logged if the data quality policy is run by a:
    • Data quality assessment.
    • Connectivity Studio data import.

You can review and process the logged validation rule execution results.

For each logged validation rule execution result, you can:

  • Set the review status.
  • Open the source record to make changes, if desired.

The validation status of a logged validation rule execution result can be:

  • Success: The validation rule is met.
  • Failed: The validation rule is not met and results in an error or warning.
  • Skipped: The validation rule is not executed because its conditions are not met.

Review and edit source record

 		Data quality

As a data owner, you can review and process data quality assessment results.

For each record with errors or warnings, you can open the source record to review the error or warning and, if desired, make changes.

Usually, you review and change the source record in case of a validation rule error or warning.

Review and merge duplicates - Basic matching

 		Data quality

A duplicate check rule of type 'Basic matching' checks if the data doesn't already exist in the table, based on a combination of table fields.

Duplicate check rules of type 'Basic matching' are applied automatically when you manually enter or update a record.

If merging duplicates is enabled for basic matching, and duplicates are found, you can compare the duplicates and merge field values. To enable merging duplicates is for basic matching, in the Data quality studio parameters, on the General tab, set the Enable duplicate check using basic matching field to Yes.

This topic explains how to handle possible duplicates that are found if merging duplicates is enabled for basic matching.

Review and merge duplicates - Quality assessment

 		Data quality

As a data owner, you can review and process data quality assessment results.

For each record with possible duplicates, you can:

  • Review the found duplicates.
  • Merge field values from the duplicate records to a chosen master record.
  • Manually delete undesired duplicate records.

Both basic and fuzzy duplicate checks are done by the quality assessment. You can review and merge found duplicates for both duplicate check types in the same way.

Run data quality assessment

 		Data quality

You can run a data quality assessment to examine the quality of existing data. To run a data quality assessment, you use data quality assessment projects.

You can do a data quality assessment, for example, if you:

  • Have implemented a change in the Data quality studio configuration and want to make sure existing data is in line with this change.
  • Have imported data without applying data quality policies.
  • Want to periodically check the quality level of your data.

Set up action rule conditions

 		Data quality

For each action rule, you can set up conditions. The action rule is only applied if the conditions are met.

To define an action rule condition, you use a query. The action rule is applied only to the records that are found by the query.
You can use these types of queries:
  • Inquiry: Use the standard D365 F&SCM Inquiry (SysQueryForm) page to define the records to which the action rule applies. The inquiry only queries the table as defined for the action rule.
  • Dynamic query: Use a dynamic query to define the records to which the action rule applies. Before you can set up a condition with a dynamic query, set up the dynamic query to be applied. You can use a dynamic query, for example, to query other tables than the action rule table, to define table relations, and to have only one record returned.
You can set up several conditions, using different query types.

Set up action rule of type Custom

 		Data quality
Use action rules to set field values in D365 F&SCM.
To set a field value, you can use several action types. This topic explains how to set up an action rule of type Custom.
Use a custom action rule to set a field value using a custom class.
Before you can set up a custom action rule, ensure that the custom class is developed.

Set up action rule of type Data query

 		Data quality

Use action rules to set field values in D365 F&SCM.
To set a field value, you can use several action types. This topic explains how to set up an action rule of type Data query.

Use a data query action rule to find a value using a query and set the found value in the target field of the action rule.
You can use these types of queries:
  • Inquiry: Use the standard D365 F&SCM Inquiry (SysQueryForm) page to define the query that is used to find the value to be set in the target field. The inquiry only queries the table as defined for the action rule.
  • Dynamic query: Use a dynamic query to define the query that is used to find the value to be set in the target field. Before you can set up an action rule with a dynamic query, set up the dynamic query to be applied. You can use a dynamic query, for example, to query other tables than the action rule table, to define table relations, and to have only one record returned.

Set up action rule of type Fixed value

 		Data quality
Use action rules to set field values in D365 F&SCM.
To set a field value, you can use several action types. This topic explains how to set up an action rule of type Fixed value.
Use a Fixed value action rule to set a pre-defined fixed value in the target field of the action rule.

Set up action rule of type Number sequence

 		Data quality

Use action rules to set field values in D365 F&SCM.
To set a field value, you can use several action types. This topic explains how to set up an action rule of type Number sequence.

Use a number sequence action rule to set the value of the target field of the action rule based on a number sequence.

Set up action rule of type Transformation list

 		Data quality

Use action rules to set field values in D365 F&SCM.

To set a field value, you can use several action types. This topic explains how to set up an action rule of type Transformation list.
Use a transformation list action rule to find a value using a transformation list and set the found value in the target field of the action rule.
Before you can set up a transformation list action rule, set up the transformation list to be used.
Example:
You can set up a transformation list to transform country codes into the relevant currencies for these countries. In this case, the source field is a country field, and the target field is a currency field. If the country is US, and the transformation list action rule is applied, the currency is set to USD.

Set up action rule of type Web service

 		Data quality

Use action rules to set field values in D365 F&SCM.

To set a field value, you can use several action types. This topic explains how to set up an action rule of type Web service.
Use a web service action rule to get a field value from another internal or external source, using a web service, and set this value in the target field of the action rule.
Before you can set up a web service action rule, set up the web service to be used.
You can set up a web service action rule in several ways. For a parameter of type:
  • Request, you can use the default parameter values as defined for the used web service. No specific parameter setup is required.
  • Request, you can override a default parameter value for a request parameter. To do so, you can use one of these types: Field, Code, or Fixed value. Each type requires specific settings. Explained in subtasks 13-14.
  • Response, you can use the values, as returned by the web service, to set several values in D365 F&SCM. Explained in subtask 15.
  • Response, you can process a response value, as returned by the web service, before you use it to set a value in D365 F&SCM. Explained in subtask 16.

Set up condition table mapping

 		Data quality

On data quality policy rules, you can define conditions. If you define a condition and the applicable table for the rule is a date-effective or inheritance table, additional setup is required. The applicable table for the rule must be mapped to a related staging table or temporary table. 

For example, if the applicable table for the rule is the:

  • LogisticsPostalAddress (inheritance) table, set up a mapping to the CustomerPostalAddressStaging table.
  • DirPartyTable (date-effective) table, set up a mapping to the DirPartyStaging table.

Set up configurable lookup - Dynamic query

 		Data quality

You can use a dynamic query to define the possible options of a configurable lookup. To do so, set up a configurable lookup of type 'Dynamic query'.

Set up configurable lookup - User defined list

 		Data quality

You can set up a configurable lookup of type 'User defined list' to manually define the possible options in a field.

Set up configurable lookup - Web service

 		Data quality

You can use a web service to define the possible options of a configurable lookup. To do so, set up a configurable lookup of type 'Web service'.

Set up data pattern

 		Data quality
You can use a data pattern validation rule to validate if a field value matches a defined pattern.
To set up a data pattern validation rule, first set up the data pattern to be applied. Define the data pattern, using a regular expression.
You can, for example, set up data patterns to validate:
  • Email addresses
  • Website URLs
  • VAT numbers
For more information on regular expressions, refer to Regular expression.

Set up data quality assessment project

 		Data quality

You can use a data quality assessment project to examine the quality of existing data.

You can do a data quality assessment, for example, if you:

  • Have implemented a change in the Data quality studio configuration and want to make sure existing data is in line with this change.
  • Have imported data without applying data quality policies.
  • Want to periodically check the quality level of your data.

In a data quality assessment project, you can apply:

  • All validation rules and duplicate check rules of a data quality policy.
  • Specific validation rules or duplicate check rules of a data quality policy.

Set up 'Data quality' organization hierarchy purpose

 		Data quality You can apply a data quality policy version to (a part of) an organization hierarchy. Before you can do so, assign the desired organization hierarchy to the 'Data quality' organization hierarchy purpose.

Set up duplicate check - Basic matching

 		Data quality

Use a basic duplicate check to ensure that no (almost) similar records exist in a table.

To set up a duplicate check rule you can use a duplicate check of type Basic matching. If the desired Basic matching duplicate check does not exist, set up a new Basic matching duplicate check.

For each Basic matching duplicate check, define which combination of table fields is checked on duplicate values. So, the combination of field values must be unique in the table. Only one record can have this combination of field values.

Set up duplicate check - Fuzzy matching

 		Data quality

Use a fuzzy duplicate check to ensure that no (almost) similar records exist in a table.

To set up a fuzzy duplicate check rule, use a duplicate check of type Fuzzy matching. If the desired Fuzzy matching duplicate check does not exist, set up a new Fuzzy matching duplicate check.

A Fuzzy matching duplicate check checks for duplicates applying fuzzy logic. A Fuzzy matching duplicate check compares, for a selected record, several field values with the values of the same fields of other records. Based on the comparison, a duplicate score is calculated.

On the duplicate check, you define:

  • Which dynamic query is used. The dynamic query defines the records that are checked for duplicate values and the fields that can be checked for duplicate values.
    In the dynamic query, the first defined table must be the main table, on which you want to check for duplicates. This table must be the same table that you define in the duplicate check header, in the Table name field.
    A form can use several related tables. In this case, in the dynamic query, use data entities for the next table records. For each of the data entity table records, define the applicable parent. Use the data entities to select the fields that you want to check for duplicates.
  • The fields which values are checked for duplicates. You only can use fields that are defined in the dynamic query.
  • For each field, the weightage. The weightage expresses the importance of a duplicate value. Express the weightage in a number (with or without decimals). Define the weightage number in such a way that the importance is expressed compared to the other fields. If you do not define a weightage for a field, the field value is not checked for duplicates.
  • The threshold that is used to indicate if a record is a possible duplicate. The threshold is expressed in a percentage. Only if the calculated duplicate score for a record is equal to or higher than the threshold, a record is marked as possible duplicate.
    The duplicate score is calculated in this way: [Weightage sum of fields with duplicates] / [Total weightage sum] * 100%

Example:

Duplicate check on CustTable

Threshold: 50%

Table name Datasource name Field Field label Weightage
CustTable CustTable AccountNum Customer account  
CustCustomerV3Entity CustCustomerV3Entity AddressStreet Street 1
CustCustomerV3Entity CustCustomerV3Entity AddressZipCode ZIP/postal code 1
CustCustomerV3Entity CustCustomerV3Entity OrganizationName Organization name 6
CustCustomerV3Entity CustCustomerV3Entity PrimaryContactEmail Primary email 3
CustCustomerV3Entity CustCustomerV3Entity PrimaryContactPhone Primary phone 3

Calculation examples:

  • Duplicate values exist in the Primary email field and in the Primary phone field. The duplicate score is: 6 / 14 * 100 = 42,86. The record is not reported as possible duplicate.
  • Duplicate values exist in the Organization name field and the ZIP/postal code field. The duplicate score is: 7 / 14 * 100 = 50. The record is reported as possible duplicate.

Set up duplicate check rule

 		Data quality

Use a duplicate check rule to check if the data doesn't already exist in D365 F&SCM.

Before you can set up a duplicate check rule, set up the duplicate check to be applied.

You can apply these types of duplicate checks:

  • Basic matching: Checks which combination of table fields has duplicate values. So, the combination of field values must be unique in the table.
  • Fuzzy matching: Checks for duplicates applying fuzzy logic. A fuzzy duplicate check compares values from several fields. Based on the comparison, a duplicate score is calculated.

Set up duplicate check rule conditions

 		Data quality

For each duplicate check rule, you can set up conditions. The duplicate check rule is only applied if the conditions are met.

To define a duplicate check rule condition, you use a query. The duplicate check rule is applied only to the records that are found by the query.
You can use these types of queries:
  • Inquiry: Use the standard D365 F&SCM Inquiry (SysQueryForm) page to define the records to which the duplicate check rule applies. The inquiry only queries the table as defined for the duplicate check rule.
  • Dynamic query: Use a dynamic query to define the records to which the duplicate check rule applies. Before you can set up a condition with a dynamic query, set up the dynamic query to be applied. You can use a dynamic query, for example, to query other tables than the duplicate check rule table, to define table relations, and to have only one record returned.
You can set up several conditions, using different query types.

Set up enrichment line of type Custom

 		Data quality

For an enrichment rule, use an enrichment rule line to set a field value. To set a field value, you can use several value types. Use the value type Custom to set a field value using a custom class.

Before you can set up a custom enrichment rule line, ensure that the custom class is developed.

You can add enrichment lines in these ways:

  • Manually.
  • Select the target fields from a page.

Set up enrichment line of type Data query

 		Data quality
For an enrichment rule, use an enrichment line to set a field value. To set a field value, you can use several value types. Use the value type Data query to find a value using a query and set the found value in the target field of the enrichment rule line.

In a data query enrichment rule line, use a dynamic query to define the query that is used to find the value to be set in the target field. Before you can set up an enrichment rule line with a dynamic query, set up the dynamic query to be applied. You can use a dynamic query, for example, to query other tables than the enrichment rule table, to define table relations, and to have only one record returned.

You can add enrichment lines in these ways:

  • Manually.
  • Select the target fields from a page.

Set up enrichment line of type Enable/Disable

 		Data quality

For an enrichment rule, use an enrichment rule line to set a field value or behavior.

To set a field value or behavior, you can use several value types. Use the value type Enable/Disable to enable or disable a field based on the value of another field.

Set up enrichment line of type Fixed value

 		Data quality
For an enrichment rule, use an enrichment rule line to set a field value.
To set a field value, you can use several value types. Use the value type Fixed value to set a pre-defined fixed value in the target field of the enrichment rule line.

You can add enrichment lines in these ways:

  • Manually.
  • Select the target fields from a page.

Set up enrichment line of type Number sequence

 		Data quality

For an enrichment rule, use an enrichment rule line to set a field value. To set a field value, you can use several value types. Use the value type Number sequence to set the value of the target field of the enrichment rule line based on a number sequence.

You can add enrichment lines in these ways:

  • Manually.
  • Select the target fields from a page.

Set up enrichment line of type Source field

 		Data quality

For an enrichment rule, use an enrichment rule line to set a field value.

To set a field value, you can use several value types. Use the value type Source field to set the value of the target field value of the enrichment rule line based on the value of a field in the source datasource.

You can add enrichment lines in these ways:

  • Manually.
  • Select the target fields from a page.

Set up enrichment line of type Transformation list

 		Data quality

For an enrichment rule, use an enrichment line to set a field value. To set a field value, you can use several value types. Use the value type Transformation list to find a value using a transformation list and set the found value in the target field of the enrichment rule.

Before you can set up a transformation list enrichment line, set up the transformation list to be used.

You can add enrichment lines in these ways:

  • Manually.
  • Select the target fields from a page.

Example:

You can set up a transformation list to transform country codes into the relevant currencies for these countries. In this case, the source field is a country field, and the target field is a currency field. If the country is US, and the transformation list enrichment rule is applied, the currency is set to USD.

Note: You can set up a transformation list enrichment line only if the enrichment rule event is 'Source field modified'.

Set up enrichment line of type Web service response

 		Data quality

For an enrichment rule, use an enrichment rule line to set a field value. To set a field value, you can use several value types. Use the value type Web service response to get a field value from another internal or external source, using a web service, and set this value in the target field of the enrichment rule line.

Before you can set up a web service response enrichment rule line:

  • Set up the web service to be used.
  • Define the web service that applies to the enrichment rule. Also define the desired request parameter setup. 

Set up a separate enrichment rule line for each target field that must be set using a web service response parameter.

You can add enrichment lines in these ways:

  • Manually.
  • Select the target fields from a page.

Note: You can set up a enrichment rule line of type 'Web service response' only if the enrichment rule event is 'Source field modified' or 'Save record'.

Set up enrichment rule conditions

 		Data quality

For each enrichment rule, you can set up conditions. The enrichment rule is only applied if the conditions are met.

To define an enrichment rule condition, you use a query. The enrichment rule is applied only to the records that are found by the query.
You can use these types of queries:
  • Inquiry: Use the standard D365 F&SCM Inquiry (SysQueryForm) page to define the records to which the enrichment rule applies. The inquiry only queries the table as defined for the enrichment rule.
  • Dynamic query: Use a dynamic query to define the records to which the enrichment rule applies. Before you can set up a condition with a dynamic query, set up the dynamic query to be applied. You can use a dynamic query, for example, to query other tables than the enrichment rule table, to define table relations, and to have only one record returned.
You can set up several conditions, using different query types.

Set up organization assignment for Data quality policy

 		Data quality

By default, a data quality policy version is applied to all organizations as defined in the current D365 F&SCM environment. If a data quality policy version must be applied only to specific organizations, set up the organization assignment.

To set up the organization assignment, you can use legal entities or organization hierarchies. Before you can use an organization hierarchy, assign the desired organization hierarchy to the 'Data quality' organization hierarchy purpose.

Set up phonetic search rules

 		Data quality

On duplicate checks of type 'Fuzzy matching', you can apply phonetic search algorithms. You can use a phonetic search algorithm to check on duplicate names that sound similar, for example, John and Jon.

To apply a phonetic search algorithm to a duplicate check, set up a phonetic search rule and link it to a field in a duplicate check.

The supported phonetic search algorithm is Metaphone. You can apply these versions of the Metaphone algorithm:

  • Double metaphone
  • Metaphone 3

Advanced setup

You are advised to start applying phonetic search with the basic setup, that is with the selected Phonetic search algorithm.

Based on testing and experience, you can finetune the phonetic search rule setup by defining:

  • A maximum length: The maximum number of characters for a phonetic search key. The shorter a phonetic search key is, the fuzzier the duplicate check result is.
  • Words to be ignored: You can define the words for which you do not want to create a phonetic search key.

Set up reason codes

 		Data quality

If you do data quality assessments, for an error or warning, you can set the status to 'Accept deviation'. In this case, it is important to explain why you accept the deviation. You can do so, by selecting a reason code.

This topic explains how to set up the reason codes.

Set up secured values

 		Data quality

You can set up secured values to store secrets at a central place in Data quality studio. You can , for example, set up a secured value for a license key or access token.

Benefits of using secured values are:

  • One place to maintain secrets. For example, you can use a secret in several web services. If the secret expires, you only update the secured value instead of updating the secret separately for each applicable web service.
  • Secrets are not shown or visible where these are applied using secured values.

In Data quality studio, you can use secured values as a parameter for web services.

Set up target datasource

 		Data quality

For an enrichment rule, you can set a field value in a target table that is different from the source table.

As a target  table, you can only use a table that is related to the source table.

Set up the tables that you want to use as target table for the enrichment rule.

Note: On creation of an enrichment rule, automatically, the source table is added as target table.

Set up transformation list

 		Data quality
To set up a transformation list action rule, a transformation list is required. If the desired value transformation does not exist, set up a new transformation list or add the transformation to an existing transformation list.
For each transformation list, set up the value transformations. For each value transformation, define the source value and the target value.
Example:
You can set up a transformation list to transform country codes into the relevant currencies for these countries. In this case, the source field is a country field, and the target field is a currency field. If the country is US, and the transformation list action rule is applied, the currency is set to USD.

Set up validation rule conditions

 		Data quality
For each validation rule, you can set up conditions. The validation rule is only applied if the conditions are met.
To define a validation rule condition, you use a query. The validation rule is applied only to the records that are found by the query.
You can use these types of queries:
  • Inquiry: Use the standard D365 F&SCM Inquiry (SysQueryForm) page to define the records to which the validation rule applies. The inquiry only queries the table as defined for the validation rule.
  • Dynamic query: Use a dynamic query to define the records to which the validation rule applies. Before you can set up a condition with a dynamic query, set up the dynamic query to be applied. You can use a dynamic query, for example, to query other tables than the validation rule table, to define table relations, and to have only one record returned.
You can set up several conditions, using different query types.

Set up validation rule message

 		Data quality You can define a message that is shown if the validation rule is not met.
If no validation rule message is defined, the default message is shown if the validation is not met.
In case of a data pattern validation, the applicable data pattern can also have a message defined. In this case, the message priority is:
  1. If a validation rule message is defined, this message is shown.
  2. If no validation rule message is defined, the data pattern message is shown.
  3. If no data pattern message is defined, the default message is shown.
You can also enter a description of the validation rule.
You can translate both the validation message and the validation rule description.

Set up validation rule of type Configurable lookup - Dynamic query

 		Data quality

Use validation rules to check if the data is in line with the defined standards.
To validate data, you can use several validation types. This topic explains how to set up a configurable lookup validation rule with a configurable lookup of type 'Dynamic query'. 

Use a:

  • Configurable lookup validation to add a custom lookup to a field and to validate if the field value matches a value as defined by the configurable lookup.
  • Configurable lookup of type 'Dynamic query' to apply a dynamic query to define the possible options of a field.

You can add validation rules in these ways:

  • Manually.
  • Select the fields to be validated from a page.

Note:

  • If the field to which the configurable lookup is applied, already has a lookup, the configurable lookup overwrites the existing lookup.
  • Before you can set up a configurable lookup validation rule, set up the configurable lookup to be used.

Set up validation rule of type Configurable lookup - User defined list

 		Data quality

Use validation rules to check if the data is in line with the defined standards.
To validate data, you can use several validation types. This topic explains how to set up a configurable lookup validation with a configurable lookup of type 'User defined list'. 

Use a:

  • Configurable lookup validation to add a custom lookup to a field and to validate if the field value matches a value as defined by the configurable lookup.
  • Configurable lookup of type 'User defined list' to manually define the possible options in a field.

You can add validation rules in these ways:

  • Manually.
  • Select the fields to be validated from a page.

Note:

  • If the field to which the configurable lookup is applied, already has a lookup, the configurable lookup overwrites the existing lookup.
  • Before you can set up a configurable lookup validation rule, set up the configurable lookup to be used.

Set up validation rule of type Configurable lookup - Web service

 		Data quality

Use validation rules to check if the data is in line with the defined standards.
To validate data, you can use several validation types. This topic explains how to set up a configurable lookup validation rule with a configurable lookup of type 'Web service'. 

Use a:

  • Configurable lookup validation to add a custom lookup to a field and to validate if the field value matches a value as defined by the configurable lookup.
  • Configurable lookup of type 'Web service' to get the possible options of a field from a web service.

You can add validation rules in these ways:

  • Manually.
  • Select the fields to be validated from a page.

Note:

  • If the field to which the configurable lookup is applied, already has a lookup, the configurable lookup overwrites the existing lookup.
  • Before you can set up a configurable lookup validation rule, set up the configurable lookup to be used.

Set up validation rule of type Custom

 		Data quality

Use validation rules to check if the data is in line with the defined standards.
To validate data, you can use several validation types. This topic explains how to set up a validation rule of type Custom.

Use a custom validation rule to validate a field value using a custom validation class.
Before you can set up a custom validation rule, ensure that the custom validation class is developed.

You can add validation rules in these ways:

  • Manually.
  • Select the fields to be validated from a page.

Set up validation rule of type Data pattern

 		Data quality

Use validation rules to check if the data is in line with the defined standards.
To validate data, you can use several validation types. This topic explains how to set up a validation rule of type Data pattern.

Use a data pattern validation rule to validate if a field value matches a defined pattern.
Before you can set up a data pattern validation rule, set up the data pattern to be applied.

You can add validation rules in these ways:

  • Manually.
  • Select the fields to be validated from a page.

Set up validation rule of type Mandatory or Blank

 		Data quality

Use validation rules to check if the data is in line with the defined standards.

To validate data, you can use several validation types. This topic explains how to set up validation rules of these types:
  • Mandatory: Makes it mandatory to fill the defined field.
  • Blank: Validates if no value is entered in a field.

You can add validation rules in these ways:

  • Manually.
  • Select the fields to be validated from a page.

Set up validation rule of type Range expression

 		Data quality
Use validation rules to check if the data is in line with the defined standards.
To validate data, you can use several validation types. This topic explains how to set up a validation rule of type Range expression.
Use a range expression validation rule to validate if a field value is within a defined range. To define a range expression, you can use a:
  • Fixed value or a range of values using the advanced query syntax.
  • Static method that defines the range. For example, you can use the SysQueryRangeUtil class to apply advanced date queries.
For more information on how to define ranges in the Range expression field, refer to Advanced filtering and query syntax.

You can add validation rules in these ways:

  • Manually.
  • Select the fields to be validated from a page.

Set up validation rule of type Web service

 		Data quality

Use validation rules to check if the data is in line with the defined standards.
To validate data, you can use several validation types. This topic explains how to set up a validation rule of type Web service validation type.

Use a web service validation rule to validate if a field value matches a value in another internal or external source, using a web service.
You can, for example, use a web service validation rule to validate an address, email, or VAT number.
Before you can set up a web service validation rule, set up the web service to be used.

You can add validation rules in these ways:

  • Manually.
  • Select the fields to be validated from a page.
You can set up a web service validation rule in several ways. For a parameter of type:
  • Request, you can use the default parameter values as defined for the used web service. No specific parameter setup is required.
  • Request, you can override a default parameter value for a request parameter. To do so, you can use one of these types: Field, Code, or Fixed value. Each type requires specific settings. Explained in subtasks 10-12.
  • Response, you can use the values, as returned by the web service, to validate a value in D365 F&SCM. Explained in subtask 13.
  • Response, you can process a response value, as returned by the web service, before you use it to validate a value in D365 F&SCM. Explained in subtask 14
 

Set up web service for Data quality studio

 		Data quality You can use a web service in:
  • A validation rule to validate if a field value matches a value in another internal or external source.
  • An action rule to fill a field value with a value from another internal or external source.
To set up a web service validation rule or action rule, first set up the web service to be used.
You can find most of the required values to be set for the web service in the technical documentation as shared by the web service provider.

Start quality assessment results review

 		Data quality

As a data owner, you can review and process data quality assessment results.

For each run quality assessment project, all related records with at least one warning or error are shown in the Quality assessment exceptions log.

For each error or warning message record, set the review status.

Synchronize phonetic search keys

 		Data quality

A fuzzy duplicate check with phonetic search rules applied, uses phonetic search keys to search for possible duplicate values. For better performance of duplicate checks, the phonetic search keys for existing data must be generated and synchronized regularly. The synchronized phonetic search keys are stored in the DQSPhoneticKey table.

To have up-to-date phonetic search keys, you are advised to synchronize the phonetic search keys several times per day. Regular synchronization of phonetic search keys is required due to:

  • New or changed data on which fuzzy duplicate checks are done.
  • New or changed fuzzy duplicate checks phonetic search setup.

On synchronization, phonetic search keys are created for:

  • Active fuzzy duplicate checks.
  • Only for the table fields with phonetic search setup.

The phonetic search keys are created considering the setup of the applicable phonetic search rule:

  • Phonetic search algorithm.
  • Maximum phonetic search key length.
  • Words to be ignored.

Test web service configuration

 		Data quality

You can test a web service configuration.

When you test the web service configuration, this is tested:

  • Connection to the web service.
  • Request that is sent to the web service.
  • Response that is received from the web service.

View quality assessment exceptions log

 		Data quality

As a data quality officer, you can review and process data quality assessment results. A data quality officer can be a:

  • General data quality officer who can review and process quality assessment results for all companies.
  • Company-specific data quality officer who can review and process quality assessment results only for an assigned company.

You can review the full unfiltered list of data quality assessment results.

Using this list, you can:

  • For each error or warning, open the source record to make the desired changes to the record.
  • If a Data entry workflow integration is set up for the quality assessment project, for each record with errors or warnings, you can start a data entry workflow to make the desired changes to the record.
    Note: You can start only one data entry workflow for each record with errors or warnings. A record can fail several validations or duplicate checks. For each failed validation or duplicate check, a separate entry is created in the Quality assessment exceptions log. So, if you have started a data entry workflow for an entry, you cannot start another data entry workflow for another entry of the same record.
  • If possible duplicate records are found, review the possible duplicate records.
  • If possible duplicate records are found, mark which record is the master record and merge field values from the other duplicate records to the master record.
  • Set the review status for one or several errors and warnings at once.

Provide feedback