Home / Product Setup & Usage / Configuration

Deduplication

By default, Transend Migration Console tracks all email, calendar, contacts and tasks that it migrates by writing select metadata to a dedup file using an MD5 Hash algorithm. Upon a rerun of those items, Transend Migration Console reads from the source and checks the user.dup file to see if the item exists in the dedup file. If an exact match is identified in the dedup file (meaning it was already migrated), the item is skipped during the rerun. If an exact match does not exist in the dedup file, the item is migrated during the rerun.

The user.dup files are found in the following directory:

  • C:\ProgramData\Transend\tmcd\artifacts\$sourceIdentifier-dedup.zip.

The items are tracked by the default Dedup Criteria of BSDNRFHU.

  • B = Message Body
  • S = Subject
  • D = Date
  • N = Sender
  • R = Recipient List
  • F = Folder
  • H = Attachment Header
  • U = Unique ID ($MessageID)

The default setting can be edited (but it’s not recommended) by navigating to Configuration Settings > Optional > Advanced Options.

Alt text

Important Notes:

  • “X” can be added into the Deduping Criteria field to force use of the Notes Unique ID (e.g. the Notes UNID) for deduplication. The value “U”, which is on by default, forces use of the internet message ID. For either option, if the specified ID is blank in the source item, the other value will be used. To specify X remove U, or to specify U remove X.

  • We recommend instructing users to “cleanup” their mailboxes prior to starting the email migration. If an item is migrated to the target mailbox and then the user deletes that item in the source, the item remains in the target mailbox.

  • An item will be migrated again if the item changes based on the Dedup Criteria used during the migration.

  • If a user moves an item to another folder, with the F for Folder defined in the Dedup Criteria, then the item will be migrated again to the new folder on a subsequent run. The originally migrated item in the first folder remains in the target mailbox, and then when the user moved it to a new folder that triggers Transend to migrate the item again since the folder does not match in the user.dup file.

  • If a user deletes a source item that has already been migrated, the item remains in the target mailbox.

  • Do not change the Dedup Criteria once you have started production migrations, this can cause items to be duplicated.

  • Do not delete the user.dup file in C:\ProgramData\Transend\tmcd\artifacts or any subsequent migration jobs will perform a full migration and duplicate data.

  • Do not run the same user migration on multiple Console machines, as this will duplicate data.

  • Clicking the Include previously migrated items box will cause all items that are in the user.dup file to be migrated again. This can be useful if you have cleared out the target mailbox for any reason and wish to rerun a full migration of all data.

    Alt text