Web Development

Data Migration Between Systems: A Safe Process

A data migration is only finished when the new system is trusted. Here is a step-by-step process that keeps records intact and gives you a way back.

Quick answer: A safe data migration follows a repeatable path: inventory what you have, map fields on purpose, clean before you move, migrate in controlled batches, validate totals and spot checks, then cut over with a rollback plan. Skipping mapping and validation is how customer records, invoices and history get lost or duplicated.

Teams migrate data when they change CRM, replace an ERP module, merge companies, move hosts, or reconnect a website to a new backend. The technology varies (CSV export, ETL tool, API sync, database replica). The failure modes look the same: wrong field mapping, silent truncation, timezone surprises, and nobody who can prove the new system matches the old one.

This checklist is system-agnostic. Use it whether you are moving contacts into a SaaS CRM or shifting order history into a new ecommerce stack.

When this checklist applies

  • CRM to CRM (for example old desk tool to a modern sales platform)
  • Accounting or ERP cutovers
  • Website platform changes that carry users, products or posts
  • Combining two company databases after an acquisition
  • Moving from spreadsheets that became a “system” into proper software

If you are only connecting two live systems for ongoing sync, that is integration, not a one-off migration. See API integration for that pattern. If you are modernising a legacy application entirely, pair this with legacy system modernization planning.

Phase 0: Define success and ownership

Write down:

  • Which systems are source and target
  • Which record types move (customers, orders, products, files, users)
  • What “done” means (for example: all open invoices match within an agreed tolerance; every active customer can log in; report X matches within 0.1%)
  • Who can approve cutover
  • Who is on call for the first week after go-live

Without an owner, migrations stall in endless “almost ready” states.

Phase 1: Inventory the source

Catalogue objects and volumes. Row counts per table or export. Growth rate so weekend windows stay realistic.

Find hidden dependencies. Attachments, notes, custom fields, archived statuses, soft-deleted rows that still matter for audit.

Document identifiers. What is the primary key today? Will you preserve it, or create new IDs and keep a mapping table? Preserving IDs makes support easier; new IDs are sometimes required by the target.

Capture credentials and rate limits. API migrations fail in the night when someone hits a throttle no one read about.

Phase 2: Map fields on purpose

Build a spreadsheet with one row per field:

Source field Type Sample Target field Transform Notes

Call out:

  • Enumerations that do not match (status values, country codes)
  • Name fields split or merged differently
  • Currencies and decimal precision
  • Timezones and stored UTC versus local time
  • Free-text fields that contain data which ought to be structured

Unmapped source fields need an explicit decision: drop, archive to a side store, or extend the target.

Phase 3: Clean before you move

Migration is a terrible moment to discover that 30 percent of emails are invalid.

  • Deduplicate with clear rules (which record wins)
  • Normalise phones, countries, postcodes where the target validates strictly
  • Fix or quarantine records that cannot meet target required fields
  • Agree how to handle historical junk you will not import

Cleaning in the source (or in a staging database) is easier than repairing production after cutover.

Phase 4: Choose the movement method

CSV or flat files. Simple, visible, easy to diff. Painful for relationships and large binaries.

API extract and load. Respects business rules in the target. Slower; watch rate limits and partial failures.

Database ETL. Fast for large volumes; easier to bypass validation by mistake. Use when you understand both schemas.

Vendor import tools. Many SaaS products ship importers. Use them when they cover your objects; do not force a tool that mangles relationships.

Often the real design is hybrid: API for live entities, bulk load for history, files for attachments.

Phase 5: Build a staging rehearsal

Never rehearse only in slides.

  1. Stand up a target sandbox
  2. Run the full pipeline with production-like volumes if you can
  3. Time each step
  4. Log errors to a file you actually read
  5. Repeat until error rates are understood and acceptable

Rehearsal also trains the team on the runbook so cutover night is not the first time someone runs the script.

Phase 6: Validate like an auditor

Validation is not “the import said success.”

Record counts. Per object, compare source versus target (accounting for deliberate filters).

Financial totals. Sum orders, payments, outstanding balances.

Referential checks. Every order line points at a real order and product.

Spot checks. Pull 20 random customers and compare field by field. Pull the top 20 by revenue too; VIPs catch issues sampling misses.

Login and app tests. Reset password flows, permissions, saved searches, documents.

Reports. Rebuild two or three critical reports in the new system and compare.

Phase 7: Cut over with a rollback plan

A typical cutover pattern:

  1. Freeze or queue writes on the source (maintenance window)
  2. Final incremental sync
  3. Validation gate (go / no-go)
  4. Point apps, websites and integrations at the target
  5. Monitor error logs and queues
  6. Keep the source read-only for an agreed fallback period

Rollback means more than a database snapshot. DNS, API keys, scheduled jobs and staff bookmarks all need a reverse plan. Say the rollback decision time out loud before you start (for example: “We decide by 05:00 or we proceed and fix forward”).

Phase 8: Hypercare

The week after migration is part of the project:

  • Daily count comparisons
  • A triage channel for staff
  • A list of known gaps (and dates to close them)
  • Security review of who got broad access “just for the migration”

Then decommission credentials that were only needed for the move.

Common failure patterns

  • Migrating active records but forgetting open activities or tasks tied to them
  • Truncating long notes because the target field is shorter
  • Double-importing because a failed job was restarted without idempotency
  • Changing phone formats so SMS and dialler integrations stop matching
  • Assuming UTF-8 everywhere when one legacy export is Windows-1252

FAQ

How long should a migration take?

Small CRM moves can finish in days of work plus a short window. ERP and multi-system programmes take weeks or months, mostly in mapping and cleansing.

Can we run both systems forever?

Parallel run helps for a while. Forever means double entry and drift. Set an end date for the dual run.

Do we need consultants?

If the data is financial, highly regulated, or the team has never done a cutover, experienced help pays for itself. Simple list moves may not need it.

What about GDPR and personal data?

Know what personal data you are copying, where it will live, and whether the new processor agreements are signed. Do not keep shadow copies of production exports on laptops.

Move data deliberately

A migration succeeds when the business trusts the new system without keeping the old one open “just in case” forever. Inventory, map, clean, rehearse, validate, cut over, then hypercare. If you need a technical partner to design the pipeline and the rollback so custom systems stay reliable, see how we approach custom development.

About the author

Imran M

Leave a Reply

Your email address will not be published. Required fields are marked *

Ready to improve your website?

Let's build a faster, higher performing website.