abei is an open-source bookkeeping tool I build in my spare time. Statements arrive by email, each bank’s or wallet’s export is parsed into one shape, AI suggests a category, and a person confirms every entry before it is booked.
The first problem it ran into on import was deduplication. The same money shows up in several forms: the same statement imported twice; a WeChat Pay purchase that appears again on the statement of the bank card behind it; a transfer recorded once in each of two accounts; a refund with the same amount as the original order, going the other way.
Before designing the matching engine, I looked at how nine bookkeeping products and open-source projects handle this. What follows is a cleaned-up version of that research, done on 1 September 2026. Items marked unverified are ones I could not find public documentation for; real statement samples decide them.
The short version
Deduplication is not one yes-or-no check. It is five different relationships, each scored on its own: a duplicate from the same source, the same transaction seen from two sources, a transfer pair, a refund, and one payment split across several lines.
Products outside China solve only the first and the third. The same transaction from two sources (WeChat Pay or Alipay against the bank card) is a problem specific to China, and no mature product has solved it.
abei’s answer is one matching engine with configurable policies, where every matching decision is stored and can be rolled back. It replaces deduplication logic that used to live in three different places.
1. How existing products detect duplicates
| Product | Main signal | Time window | Amount tolerance | What happens on a match | What the user sees |
|---|---|---|---|---|---|
| Firefly III Data Importer | external_id (mapped to the External identifier column); can also compare description and notes | None, exact match | Zero | Row is not imported | Only in the import log; nothing in the ledger |
| Firefly III core | journal_meta.import_hash_v2, a sha256 of the whole row as JSON; errors out when errorIfDuplicate is set | None | Zero | Rejected with an error | The error message |
| Actual Budget | imported_id first; otherwise fuzzy matching (same amount, date within ±7 days, payee; then amount and date alone) | ±7 days | Exact amount | Updates the existing transaction (including moving its date to the imported one) instead of dropping the new row | The transaction changes in place |
| YNAB | import_id on import (format YNAB:{amount}:{date}:{occurrence}, unique per account); the UI pairs imported and manual entries as Matched | Not stated in the official docs (the community often quotes ±10 days, unverified) | Same amount | Marked Matched, still needs Approve; the user can Unmatch | An explicit match and approve flow |
| beangulp (Beancount’s import framework) | Heuristics in the similar module: date window, relative amount tolerance, and a subset check on the accounts involved | window_days=2 | epsilon=0.05, a 5% relative tolerance | Only tags the transaction __duplicate__; nothing is deleted or merged | The user decides in the text file |
| GnuCash import matcher | Bayesian scoring with three thresholds | Configurable number of days | Scored | Red, yellow and green zones: add by default, let a person choose, or clear by default | Every row can be changed in the import matcher |
| Plaid | Pending to posted is not a status change but a new transaction whose pending_transaction_id points back to the old one; the old one is reported as removed | Usually 1 to 5 business days, up to 14 | Not applicable | The consumer of the API merges them | Up to the app built on it |
| Lunch Money, Tiller | Mostly manual deduplication tools; Tiller tags each CSV import in an Import Tag column so a whole import can be deleted and redone | None | None | Manual | Fully visible |
| BeeCount and other Chinese open-source apps | A mandatory preview before import, with atomic rollback of the whole batch (unverified, no public documentation found) | — | — | — | — |
A few more observations:
- Firefly’s whole-row hash is extremely sensitive to normalisation. Change the description, a tag or a note, and the same transaction is no longer seen as a duplicate. The reverse also happens: two real purchases from the same merchant, for the same amount, on the same day, get flagged as one.
- Firefly’s community keeps running into “a deleted transaction is still treated as a duplicate on re-import” and its opposite, “it gets imported twice anyway” (see discussion #10096 and issue #5083). Deletion and the duplicate index do not share a lifecycle.
- Actual’s choice to update the existing transaction after a match is the key design decision. It accepts that entering a purchase by hand first and importing it later is the normal case, and that the imported data is the more authoritative copy.
2. Three problems specific to China
The same transaction from two sources
A purchase paid with WeChat Pay or Alipay shows up on the platform’s statement and again on the statement of the bank card linked to it. Products built elsewhere have no such concept; in Plaid, one account has one stream of transactions.
The community (Discussion #162 on deb-sig/double-entry-generator) describes two approaches:
| Approach | How it works | Pros | Cons |
|---|---|---|---|
| A clearing account | The bank side is booked as bank → clearing account, the platform side as clearing account → merchant; the clearing account should always sit at zero, which makes it self-checking | Keeps all information and can be reconciled | The account tree gets complicated, and ordinary users cannot follow it |
| Rule-based deduplication (the common choice) | Keep the platform-side record, which carries the most information, and ignore the matching bank row | Simple, and the books stay clean | Balances derived from the bank side miss a piece, so the pairing has to be recorded |
The key signals are the time to the second, the amount, and the last digits of the bank card in the platform’s payment method field. The same discussion also suggests automatic matching on timestamps to the second plus the amount.
Stateful automatic pairing, which remembers pairs and lets you undo them, is still an open gap; tools such as bean-sieve are experimenting with it (unverified).
A refund is not a duplicate
A refund has the same merchant and the same amount as the original order, in the opposite direction. The right move is to keep the reverse entry and link the two, not to delete one of them.
Alipay statements carry a successful-refund and fund-status field, so this can be decided from fields rather than guessed (unverified: field names depend on real samples).
One payment, several lines
One payment gets split across several statement lines, or several charges are combined into one deduction. No mature product handles this automatically, because subset-sum matching produces too many false positives. The conclusion: suggest it, never merge it automatically.
Also, WeChat marks top-ups, withdrawals, wealth-management transfers and credit card repayments as neutral transactions, and Alipay has a flag for entries that count as neither income nor spending. Neither belongs in income and spending totals (unverified: depends on real samples).
3. What the interface does after a match
| Level | Example | Relationship | What it must come with |
|---|---|---|---|
| Skip silently | Firefly’s importer | Same-source duplicates matched on a hard key | A view that shows what was skipped and which transaction it collided with. Firefly is the counterexample: once skipped, there is nowhere to look |
| Suggest, and wait for a person | YNAB’s Matched (still needs Approve, can be unmatched), GnuCash’s yellow zone | The same transaction from two sources, duplicates found on soft signals | Readable evidence, and one click to split them apart |
| Block, with a score | GnuCash’s red, yellow and green zones | High amounts or high uncertainty | A score that can be explained |
Undo needs two levels: the whole batch (like Tiller’s Import Tag, which removes one import in one go) and a single entry.
4. What users should be able to configure
Ordered by what users actually change:
- The time window per channel: hours for WeChat Pay and Alipay, days for banks.
- Amount tolerance: an absolute value and a relative one, whichever is larger. beangulp uses 5% relative; banks usually need an absolute value.
- Which unique key wins: the transaction number, then the merchant’s order number, then a fingerprint hash.
- Which channel is authoritative: which record is the main one when two sources disagree. The platform side by default.
- A strictness switch, like Firefly’s
strictIdCheckingor GnuCash’s three thresholds. - No fine-grained field weights. Users cannot tune them well and will only break them.
5. What abei does
One matching engine, five relationships scored separately
The input is one normalised row: channel, the time to the second, amount, counterparty, goods, payment method, business type, status, direction, the platform’s order number, the merchant’s order number, and a hash of the raw row.
| Relationship | Signal | Default action |
|---|---|---|
| Same-source duplicate | A hard key (the transaction number) matches | Skip silently and log it; if only soft signals match, wait for a person |
| Same transaction, two sources | Time to the second, amount, last digits of the card | Fold automatically when the score is high; the platform side is the main record, and the bank side is kept as the source of the funds |
| Transfer pair | Both accounts, the amount, a time window | Pair and offset automatically; anything that cannot be paired goes to an unbalanced queue |
| Refund | Opposite amount, same merchant, the refund field | Link automatically, never merge, and never count it as income |
| One payment, several lines | Subset sum | Only ever suggest, never merge automatically |
Every match produces a decision: the relationship, the transactions it points to, a score, the evidence and a suggested action. The evidence has to read line by line, such as “same order number”, “3 seconds apart”, “same card digits”.
States and rollback
A match starts as pending. It is either merged automatically or sent for review; a review ends in confirmed or rejected; and anything confirmed can still be unlinked later.
- The merge log is append-only and supports rollback of a single entry or a whole batch, borrowing the idea of Tiller’s Import Tag.
- A rejected pair is kept as a negative example, so the same pair never comes up again.
- AI only makes suggestions on items waiting for review. Matches on a hard key never go through AI.
Default thresholds
These are defaults that are still under discussion, and will be configurable per channel in the admin console.
| Relationship | Time window | Amount tolerance |
|---|---|---|
| Same-source duplicate | None (exact match on a hard key) | 0 |
| Same transaction, two sources | ±2 hours | 0 |
| Transfer pair | ±3 days | 0 |
| Refund | ±90 days | 0 |
What abei does not do
- No whole-row hash in the style of Firefly. abei uses a structured external key instead. The statement rows already carry an external key and a fingerprint that are more stable than Firefly’s hash; they move to the ledger table as they are, with a constraint that keeps the external key unique per user.
- No silent skip without a trace. Every skipped row has to show which transaction it collided with.
- No field weights exposed to users.
Sources
Firefly III
- Duplicate detection
- The whole-row hash (
import_hash_v2) - Deleted transactions and external ID duplicates
- Same external ID imported twice
- Deduplicating on custom fields
Actual Budget
YNAB
beangulp and Beancount
GnuCash
Plaid
Tiller
China-specific