Distill

Every "AI dedup" tool on the market is either a $500/month SaaS that runs once a week or a CSV-only batch job that doesn't talk to HubSpot. Distill is the demo that says here is the real plumbing — clustering, review UX, reversible writes, the merge log — running against your live CRM in under five minutes.

What rule-based dedup misses (and Distill catches):

• jane@acme.com vs jane.doe@acme.com vs j.doe@acme.com — same person, three records.
• Acme Inc vs Acme, Inc. vs ACME — same company, three records.
• +1-415-555-0100 vs 4155550100 vs (415) 555-0100 — same phone, three formats.

The pipeline:

Blocking (cheap predicate groupings: domain, phone, name+company) avoids the O(N²) all-pairs comparison entirely. Within each block, pairwise scoring blends cosine on text-embedding-3-small with rapidfuzz edit distance on (name, email-local-part, company), weighted sum. HDBSCAN finds the clusters. Reviewers walk the queue in a Filament admin: side-by-side diff card with column-wise highlighting, association rollup, confirm / skip / reject keyboard shortcuts.

Asking without writing:

That batch pipeline has a single-contact counterpart. POST /v1/contacts/check answers "would this contact be a duplicate?" before anything is created — email / name / company / phone plus a threshold, writing nothing. Deterministic matches (exact email, normalised phone, normalised company) merge with semantic neighbours from pgvector.

Every response carries a mode. If the embedding worker is slow, dead, or returns no vector, the answer degrades to exact and near-exact matches only and says so — mode: degraded plus a reason. A degraded answer rendered as a confident "no duplicates found" would be worse than an error: the caller inserts the duplicate and never learns the check was running blind. A success signal is where nobody looks, so it is the one that has to be honest.

Reversible writes are non-negotiable:

Every merge persists the full pre-merge snapshot of every involved record before calling HubSpot's merge API. Undo recreates the merged contacts via batch-create + re-associates owned deals via the Associations API. The known limitation — associations created against the winner after the merge cannot be perfectly restored — is surfaced inline in the undo confirmation modal, not buried in a footer.

Read-only by default. No silent auto-merges:

Every workspace starts with writes_enabled = false. The merge button is hidden in the UI until an admin explicitly flips the toggle with an "I understand Distill will write merges back to HubSpot when I confirm them in the review queue" confirmation. The flip is logged to the audit trail. Auto-merge is off by default; when enabled it only fires above a configurable similarity threshold (0.98 default) and is daily-capped (50 default).

Reject-pair memory:

Once a reviewer rejects a pair as "different people / intentional duplicate / needs more info / other," Distill remembers it permanently. Re-clustering after a new sync invalidates stale pending clusters via a version stamp but preserves human verdicts — the queue doesn't loop.

Stack:

• Laravel 13 + Filament 5 (review queue, audit log, HubSpot connection management)
• FastAPI Python 3.12 cluster worker (embed, score, HDBSCAN; supervisord-managed)
• PostgreSQL 16 + pgvector 0.8 (HNSW on the embedding column for in-cluster similarity refinement)
• Next.js 16 + Scalar (docs, marketing, 1,200-contact live demo, OpenAPI reference)
• Apache + php-fpm; PM2 for the docs site; supervisord for the cluster worker

Quality:

98 PHPUnit tests / 318 assertions in ~3s — covers HealthCheck, ApiKey scope + token, PhoneNormalizer, WriteGuard auto-merge guards, OAuth PKCE, EmbeddingText, ClusterRejection pair key, DemoFixtureGenerator, Block/Cluster/Merge/SyncRun constants + transitions, HubSpotConnection expiry buffer, WorkspaceMembership capabilities, ContactMirror merged-flag, route auth, DemoController, ClusterController, SyncController, MergeController, AuditController, HubSpot webhook signature verification, the POST /v1/contacts/check pre-insert path including its degraded mode, trace context, WorkspaceScope cross-tenant isolation. Plus 11 vitest tests for the TypeScript SDK, 4 pytest tests for the Python SDK, 11 pytest tests for the FastAPI worker, Playwright spec covering the load-bearing demo controls, and a k6 nightly perf gate with p95 SLOs on the clustering endpoints.

What it proves:

Same person hand-authored the OpenAPI spec, wrote the Laravel controllers and Filament admin, built the HubSpot OAuth + PKCE flow with rate-limit-aware backfill, designed the embedding + blocking + clustering pipeline, wrote the merge writer and the undo orchestrator with the association-recreation caveat surfaced honestly, built the Next.js docs and the 1,200-contact live demo, packaged the read-only TS + Python SDKs, and shipped all three processes live with atomic releases. The case for hiring me to ship a complete HubSpot-integrated dedup product instead of a CSV uploader stitched together with a fuzzy-match library.