Basics of Changes Only Functionality in HCM Extract

We mainly use HCM Extracts for the changes only and multi threaded processing which are inbuilt functionalities.

For Multi-threaded processing of HCM Extracts and how to set up, please check it by Clicking here

What is Changes Only?

While extracting data through HCM Extracts, we can configure the extract to identify the incremental changes after the previous extract runs. The processing engine generates the current status of the data, compares it with the base-lined data of the previous runs, and identifies the new as well as any modifications. This feature can identify new, corrections and updates to existing details, by comparing the data reported earlier. However it cannot handle the deletes that were done as it will not fetch that data.

Prerequisites

1) Unhide “Changes Only” Parameter. If it is not there add it by giving details as below and make display flag ‘Yes’

2) Define multi-threading details to indicate the level (preferably at the root data group) at which the processing engine needs to start the comparison. Any data above threading data group will always run as full extract (irrespective of changes only mode).
Select the unique Database Item Group (DBI) of the User Entity (UE) you linked to the data group as the multi-threading database item. Choose a unique DBI in the UE such as, Person_Id in our example and select ‘Object Actions’ as the Threading Action Type.Ex: If you are using PER_EXT_PAY_EMPLOYEES_UE, then ASSIGNMENT ID should be the threading database item.

3) Include Changes from Last Successful Run: Ensure that this is set. This is to make sure that diff is performed between present and last successful run only. If it is unchecked and and extract run fails due to some issue. And when you run the extract in changes only mode, the changes made in previous failed run are lost unless rollback is done for failed run. In short, if this parameter is set, there is no need to rollback the runs on failure to extract correct changes only data.

Other Settings that needs to be done at the attribute level under Records:

Key Attribute: Mark the unique attributes such as Person Id, Phone Id, Assignment ID as key for all the data groups. Key attribute is used to compare attributes in case for advanced changes only modes. Please ensure you mark one or more attributes. Simple Thumb rule  Primary Key of the Entity is marked as Key. In case of date effective historical entities, both Primary Key and Effective Start Date is marked as key.

Mark as Changed: Mark an attribute as “Mark as Changed” whenever you want to include the attribute in the output of the extract even though it has not changed. If an attribute is set as “Mark as Changed” and anything changes in the same record, then this attribute will be included in the output. However, if no attributes change in the record but an attribute changes in another record (for example, a parent or sibling record), then the attribute will not be included in the output.

Why Mark as Changed? Suppose if a phone number area-code changes, only phone number area-code is displayed in output. We wont know whose number is changed. If we give Person Number & Phone Number as Mark as Changed, they will be displayed in output and thus we can know whose number and which number has changed.  Simple Thumb rule,  Alternate Key of the Entity and Threading Object Alternate key (say Person Number) is marked as changed in the record. 

Always Display: Mark an attribute as “Always Display” whenever you want to include the attribute in the output of the extract even when they have not changed. If an attribute is set as “Always Display” and anything changes in the record or in the hierarchy beneath the record , then this attribute will be included in the output. So even if nothing changes in the attribute’s record, as long as something changes beneath it in the hierarchy it will be included. (Hierarchy here refers to the tree below threading DBI.)

What is difference between Mark as Changed and Always Display?

Mark as Changed is displayed in output only if there are any changes associated with that particular record. Or else it is not displayed. But if Always Display is set, it is displayed always if any change occurs in the hierarchy. Typically in our case , Person Number can be set as Always Display attribute which is present in Root/Threading Data Group. So, any change anywhere in hierarchy will pull out such information.

Exclude from Comparison: If this is checked, even if those attributes are changed these are treated as though not changed. Example, Effective Date if present in extract for reporting purpose, Effective/Run date may keep on changing , hence this needs to be marked as ‘Exclude from Compare’.

Changes Only Modes

In Short: 
Y: Compares this extract run with the previous extract runs and extracts the entire hierarchy even if single attribute has changed.
ATTRIBUTE: Includes only record attributes that have changed or Marked as Changed or Always display.
ATTRIB_OLD: Displays only record attributes that have changed or Marked as Changed or Always display. along with their previous value
BLOCK: Displays the all the data under threading data group (similar to Changes Only = Y), but, only Changed Data; Marked as Changed, Always Display data.
BLOCK_OLD: Displays the all the data under threading data group (similar to Changes Only = Y), but, only Changed Data; Marked as Changed, Always Display data with their previous values.

Baselining the Changes Only Run

If the requirement is to just baseline the run before running the changes only runs, unhide the parameter Baseline Only. Run the full extract with Baseline Only as ‘Yes’. 
Please note this will only baseline the extract and hence you will not get any output. Please hide the Baseline Only parameter after the run because this is used only once for the changes only extract lifetime in production.

Rollback a Changes Only Run

When you want to roll back a changes-only extract run, as if it was never run or in case of run failure; then you can use the ‘Roll Back Process’ Process.

FAQs on Changes Only Functionality in HCM Extracts:

How are the changes identified?
For a full extract (i.e. which does not involve changes-only processing), the extract processing engine extracts the data, archives the raw data and converts into XML, calls the BIP API for formatting and delivery.
For a changes-only extract, there are two phases in recognizing the changes.
Phase 1: This will involve performing a normal archive operation. When each of the object_action ex. Person ID (this is setup in the multi-threading details) has had its data archived, the processing engine locates the last successful object_action (ex. Person ID),
archived for the same row. A difference check is performed on the two archives and if a change is not detected that object_action and corresponding archive data is removed.
At this point we have data under all data groups for the object actions for which data has changed.
This is the end of changes only processing when run in mode ‘Changed Attributes only’ (Mode =Y).
Phase 2: For other modes (like Attribute, Attrib_Old, Block_Old), the next phase occurs during XML generation.
For each of the parent data group where changes were identified in Phase 1, we identify all children data groups and records that have changed with help of key.
On a field by field basis a direct comparison is performed and changes reported as required by the mode it is being run under.

What happens if there are multiple changes after the last extract run?
Extracts compares the current run output against the previous runs to identify changes. So the latest change would only be reported in this scenario.

How do I trial run a changes-only in production and then revert it back as if changes were not reported?
You can use the rollback feature to rollback the extract run so that the reported changes are cleaned. When you run the extract after a rollback, the extract results identify all the changes.
You can also run Extract with Delete Archive = CLOB parameter ,but, ensure you hide the parameter after your test run.

When a changes-only extract was running there was an unexpected event such as server restarting or maintenance window?
In this scenario, we recommend you rollback the error run so that changes-only behaves as expected. We have seen multiple issues arising when such an unexpected event caused full extract to be generated. So it’s strongly recommended that you roll back the error run.

When I make changes to the extract design, it resulted in a full extract, why?
Any change in the extract design (like addition of a new attribute, reordering ) impacts the changes-only feature as processing logic does a comparison and identifies differences.
When the processing logic sees the structure has changed or new attributes, it results in a full extract.

I have created an extract without defining the multi-threaded details and ran it. Now I am unable to update the multi-threaded details, why?
You need to specify the multi-threading details while creating the extract and this cannot be updated after running an extract.
Changing the multi threaded DBI, will mean a comparison with a previous extract that has become invalid, as now we will be using the new structure against the old structure. Depending on the change, it could result in a full extract or will miss some changes.

How do I enable Multi-threading in an existing Extract (that is already run)?
You need to copy the extract and add multi-threading details into the new extract.
Steps:
a) Copy the definition.
b) In the new extract, open the Root Data Group, and add threading DBI as %Person Id and threading action as Object Actions.
c) Compile all formula.

How do I check and update the threads?
Steps:
1) Navigate to FSM, Manage Payroll Process Configuration task.
Check/Modify values of below parameters: Parameter Name- Threads Determines the 2) total number of sub processes that run under the concurrent manager. Default: 1; minimum: 1 Modify value to new value such as 8.
3) If you already have value as 8 or above 8. Do not modify the value.
Note: If you have no such parameter, you need to create the value by clicking on (+) and
choosing above parameter in Lookup.

How do I determine the threading DBI for each UE? Does it need to be unique?
The treading DBI needs to be a unique item in the UE, for example any ID in that entity. 

I scheduled a changes-only extract, which should send data based on the run date. How do we make the parameter value dynamic so that it picks the run date?
When you recursively schedule extracts, use context binding to dynamically increment date parameters. The parameter details should specify Context Binding for the parameter basis, and System Date for the basis value.
For example, you might create a weekly pay extract that requires the user to enter an effective date. 
You specify the Context Binding for the parameter basis for the effective date, and System Date for the basis value.
Setting these parameters ensures that the dates the application derives from the defaulted date parameter are incremented appropriately.

I have used advanced conditions and the extract output was not as expected, why?
You may have used advanced conditions at record or attribute level to conditionally exclude records for some action condition. If these advanced functionalities are not designed correctly, it can lead to unexpected results.

Tip: If you are developing Changes only extract, make sure you enable the changes only parameter, define threading DBI and mark attributes as Key at record level.