Cleaning duplicate events without losing history
Duplicate screen_view and tap events inflate funnels and poison A/B results. A practical playbook for deduping while keeping longitudinal reports intact.
Duplicate events usually arrive from overlapping SDKs, auto-track plus manual track, or retries after network failure. Teams often "fix" the problem by renaming events and starting fresh—then wonder why year-over-year charts break.
We prefer a three-layer approach. First, stop the bleed: identify the emitting call sites and collapse to a single source of truth per user action. Second, add a client-side idempotency key or server-side dedupe window for known retry patterns. Third, create a cleaned derived event for analytics while retaining the raw stream for audit.
For historical continuity, map old and new names in a transformation layer rather than rewriting the warehouse. Dashboards should point at the cleaned model; engineering debug tools can still inspect raw payloads.
Validate with a side-by-side week: funnel conversion and unique user counts should stabilize within an expected band. If uniques drop sharply, you likely over-deduped legitimate multi-fire actions like search submits.