Notes / 2 February 2026

Comparing retention across release trains

Retention ledgers look scientific because they have a date on the column. The date is often a confession. If week 3 of a train shipped a paywall, a new onboarding, or a crashy video SDK, D7 for that cohort is not a usage trend. It is a mixture of habit and treatment.

In Retention Ledger we mark trains on the same page as the week-n table. A shaded column is enough. Teams still try to “adjust” by dropping the week, which is sometimes right and sometimes a way to hide a product decision. The rule we use: if you cannot describe the change in one sentence a director would recognise, you do not get to smooth it.

Peer comparison makes this worse. Your competitor may have shipped nothing that week. Their D7 is a different experiment (namely, none). Putting both numbers on a slide titled “category retention” is how nameless averages get born.

A cleaner move is to hold two ledgers: a product ledger that includes trains, and a benchmark ledger that only uses weeks both you and the peer set would call quiet. Quiet weeks are rarer than people admit. If you cannot find four of them in a quarter, the honest output is a shorter series, not interpolated dots.

None of this requires a warehouse audit from Softmatrixhub. It requires a calendar next to the query. Bring both to class.