Skip to content
← All writing & deep dives
Fintech & Hybrid Retrieval··10 min read

Classify AI: Scaling Financial Categorization to 50,000+ Daily Transactions

How we combined dense vector retrieval with sparse lexical search and per-company charts of accounts to cut manual bookkeeping by roughly 80%.

Ashwin RachhaAI Product Engineer / Tech Lead · Finally

1. What bank feeds actually look like

A payment to Amazon Web Services arrives as AMZN MKTP US*2X7BK9 WA. A Lyft ride is LYFT *RIDE 09-14 SAN FRANCISCO. A catering invoice isSQ *THE DAILY GRIND 94103. This is what banks send. Nobody cleans it up before it lands in a ledger.

A bookkeeper managing hundreds of client companies sorts thousands of these a month into specific general ledger accounts. It is grinding work, and the hard part is not volume. It is that the same string means different things to different companies. For a software startup, AWS belongs in cost of goods sold. For a law firm, it is internal office software.

And the stakes are asymmetric. Classify a balance-sheet transfer as an operating expense and the tax filing is wrong. Messy input, tenant-specific truth, no room for confident errors.

2. Retrieval, not generation

The obvious approach is prompting an LLM per transaction. At 50,000 transactions a day that fails twice: it costs real money, and it adds seconds of latency to every line item. But more fundamentally, generation is the wrong shape. The answer already exists in the company’s own history. The job is finding it.

So Classify AI was built as retrieval with rules on top:

  1. Merchant tokenization. Regex cleaning, store-number stripping, normalization. Deterministic, cheap, boring.
  2. Dense vector search (Pinecone). Embeddings of merchant descriptions and the company’s own past corrections, capturing intent.
  3. Sparse lexical search (Elasticsearch BM25). Exact matches on invoice numbers, SKUs, accounting codes. The string is the anchor.
  4. Chart of accounts re-ranking. Candidates filtered strictly within each client’s approved taxonomy.
  5. Confidence gating. Predictions above 92% post to QuickBooks. The rest goes to a bookkeeper.

The two retrievers earn their keep separately. Dense search finds meaning; BM25 finds the literal string. A vendor with an invoice number in the description should match on that number, not on what the description roughly means. Companies that mix both got fewer wrong answers than either alone.

System ArchitectureClassify AI Hybrid Retrieval & Execution Architecture
.excalidrawPNGSVG
Classify AI Hybrid Retrieval & Execution Architecture
Click to expand & zoom
Diagram Details: End-to-end transaction categorization pipeline: Users interact with books.finally.com; Books Backend integrates Plaid Service, Heron Service, and custom Rules Engine. Asynchronous classification tasks dispatch via Redis and Celery into the AI Server. The hybrid engine queries Key-Value caches, Postgres DB, and Pinecone vector search alongside Elasticsearch BM25, outputting normalized JSON categorizations with confidence metrics.

3. Living inside the exceptions

The pipeline was built for the tail. Recurring vendors like Slack and GitHub resolve in under 15 milliseconds from cache, and that compute headroom is what buys time for the ambiguous ones. Batch ingestion is idempotent, verified against ledger hash digests, because a duplicate posting in a ledger is not a bug you patch, it is trust you lose.

The 92% threshold is where the product thinking lived. Every point of confidence is a trade between the bookkeeper’s time and the ledger’s integrity. We chose to leave the uncertain work visible rather than automate a guess. The number of transactions routed to review was a feature you could watch, not an error we hid.

4. What changed

Manual categorization dropped by roughly 80%. A first-month close that took more than four months of manual review took about two weeks. Bookkeepers stopped tagging transactions and started working on accruals and tax strategy, which is what they were actually for.

The lesson I carry from this one: the model was never the product. The chart of accounts, the confidence gate, the review queue. That was the product. The embedding just made it fast.

Related