Home / Product Setup & Usage / Configuration

Deduplication

By default, Transend Migration Console tracks all email, calendar, contacts and tasks that it migrates by writing select metadata to a dedup file using an MD5 Hash algorithm. Upon a rerun of those items, Transend Migration Console reads from the source and checks the user.dup file to see if the item exists in the dedup file. If an exact match is identified in the dedup file (meaning it was already migrated), the item is skipped during the rerun. If an exact match does not exist in the dedup file, the item is migrated during the rerun.

The user.dup files are found in the following directory:

  • C:\ProgramData\Transend\tmcd\artifacts\$sourceIdentifier-dedup.zip.

The items are tracked by the default Dedup Criteria of BSDNRFHU.

  • B = Message Body
  • S = Subject
  • D = Date
  • N = Sender
  • R = Recipient List
  • F = Folder
  • H = Attachment Header
  • U = Unique ID ($MessageID)

The default setting can be edited (but it’s not recommended) by navigating to Configuration Settings > Optional > Advanced Options.

Alt text

Important Notes:

  • Maintain consistent migration protocols – It is recommended that you do not change migration protocols (e.g., switching between Microsoft EWS and Microsoft Graph) during an active Production migration. Differences in how protocols process and identify data can impact deduplication, potentially resulting in duplicate emails, calendar events, contacts, or tasks during subsequent migration jobs. To minimize the risk of duplication and maintain data consistency, use the same protocol throughout the entire Production migration.

  • “X” can be added into the Deduping Criteria field to force use of the Notes Unique ID (e.g. the Notes UNID) for deduplication. The value “U”, which is on by default, forces use of the internet message ID. For either option, if the specified ID is blank in the source item, the other value will be used. To specify X remove U, or to specify U remove X.

  • We recommend instructing users to “cleanup” their mailboxes prior to starting the email migration. If an item is migrated to the target mailbox and then the user deletes that item in the source, the item remains in the target mailbox.

  • An item will be migrated again if the item changes based on the Dedup Criteria used during the migration.

  • If a user moves an item to another folder, with the F for Folder defined in the Dedup Criteria, then the item will be migrated again to the new folder on a subsequent run. The originally migrated item in the first folder remains in the target mailbox, and then when the user moved it to a new folder that triggers Transend to migrate the item again since the folder does not match in the user.dup file.

  • If a user deletes a source item that has already been migrated, the item remains in the target mailbox.

  • Do not change the Dedup Criteria once you have started production migrations, this can cause items to be duplicated.

  • Do not delete the user.dup file in C:\ProgramData\Transend\tmcd\artifacts or any subsequent migration jobs will perform a full migration and duplicate data.

  • Do not run the same user migration on multiple Console machines, as this will duplicate data.

  • Clicking the Include previously migrated items box will cause all items that are in the user.dup file to be migrated again. This can be useful if you have cleared out the target mailbox for any reason and wish to rerun a full migration of all data.

    Alt text

Dynamic Deduplication

For customers using TMC v3.3 or later to migrate to Microsoft 365 using the Graph API, Google Workspace, or a compatible IMAP server, we recommend testing the optional Dynamic Deduplication setting.

Dynamic deduplication uses information in the target platform to identify previously migrated items, reducing reliance on locally stored deduplication files. This simplifies migration management and provides greater flexibility when running subsequent migration passes from a different machine or after changes to account identifiers.

The tradeoff is additional processing time, as TMC checks the target for previously migrated items during each pass.

Deduplication for Tenant-to-Tenant Migrations

When performing a tenant-to-tenant migration (such as Google Workspace to Google Workspace or Microsoft 365 to Microsoft 365) special consideration must be given to deduplication.