Multi-Tenant CMS: Serving Many Brands from One Spring Boot Codebase

We had a production site for one business. Then a second business needed a site. The lazy answer is "copy the repo and change the colours" — and now you maintain two codebases forever. Instead we converted the app into a multi-tenant CMS: one deployment, one database, many branded websites. Here's the design, the request lifecycle, and the bugs that actually bit us.

The goal

Multiple businesses, each with its own domain, theme, navigation, pages, blog, SEO and content — all served from a single Spring Boot app and configured from the database. Adding a new business should need data + DNS + an nginx vhost, not a code change or a new deployment.

The core idea: resolve the tenant from the Host header

Every request already tells you which brand it wants — in the Host header. nginx terminates TLS per domain and proxies every domain to the same app, forwarding Host $host. A servlet interceptor reads request.getServerName(), looks up the Website, and stashes it in a request-scoped ThreadLocal that the rest of the request can read.

The request lifecycle

1. nginx matches the domain's server block, proxies to the app, forwards Host $host.
2. TenantInterceptor.preHandle calls TenantResolver: normalise the host (strip www.), match a Website by host, else fall back to the default tenant. A dev-only ?tenant= override helps local testing (off in prod).
3. The resolved Website goes into TenantContext (a ThreadLocal), and a @ControllerAdvice exposes it to every view as ${currentWebsite}.
4. Controllers ask the service layer for content for that website; Thymeleaf renders the tenant's theme + content.
5. afterCompletion clears the ThreadLocal — non-negotiable on a pooled thread, or the next request could inherit the wrong tenant.

Low-level design — layers & key classes

Strict one-way dependency: Controller → Service → Repository → Entity. Controllers do HTTP only; tenant filtering lives in the repository query signatures; tenant resolution, auth and audit are cross-cutting interceptors/advice.

Sequence: a request becomes a themed page

End to end — from the Host header to rendered HTML — including the ThreadLocal cleanup that stops one request bleeding into the next.

Data isolation: the invariant that keeps tenants apart

The rule that prevents one brand leaking into another is boring and absolute: every content query takes a Website. There is no "get all services" — only findByWebsiteAndActiveTrueOrderByDisplayOrderAsc(website). A read-facade service is the single place tenant filtering lives, and admin writes re-verify row.website == selectedWebsite so a tampered id can't move a row across tenants.

Theming without forking templates

Each Website owns a Theme (logo, favicon, colours, fonts, an optional CSS file). A small Thymeleaf fragment emits CSS custom properties into :root and links the tenant stylesheet:

--color-primary, --color-secondary, --font-heading … set per tenant, consumed by the templates. Swapping a brand's look is a data change, not a deploy. (A neat gotcha: output the values unescaped — ..., not ... — or the quotes in a font-family become ' and CSS ignores them.)

One admin, many sites

A single Spring Security-protected admin runs every tenant. You log in once and pick which site you're editing from a dropdown — the selection lives in the session, deliberately independent of the host, so one operator manages all brands from any domain. Roles split ADMIN (full, incl. user management + audit log) from EDITOR (content only). Every mutating admin request is written to an audit trail by an interceptor.

Key design decisions

Gotchas we actually hit

1. Thymeleaf fragment name collision. A fragment named head silently matched the literal <head> tag instead of the parameterised fragment. Name fragments so they can't clash with HTML tags (tenantHead, seoHead).
2. SpEL and needs booleans. ${post.author and post.publishedAt} throws — those are a String and a date. Use ${x != null and y != null}.
3. ddl-auto=update can't add a NOT NULL column to a populated table (existing rows have no value). Make new columns nullable and backfill, or ship a real migration (Flyway is the grown-up answer).
4. Multipart + CSRF. The CSRF token in a multipart body isn't read cleanly; we exempted the authenticated upload endpoint rather than fight the filter ordering.
5. Volume ownership. The container runs as a non-root user, but a mounted /app/media is root-owned by default — uploads fail. chown the dir in the Dockerfile so the fresh named volume inherits the right owner.

Takeaways

1. Let the Host header pick the tenant — it's already there, and nginx just forwards it.
2. Push tenant filtering into repository method signatures so isolation can't be forgotten.
3. Always clear the ThreadLocal after the request on pooled threads.
4. Make branding, navigation, pages and SEO data, not code — that's what makes onboarding a new brand a config task.
5. Don't touch the working incumbent; grow new behaviour through the CMS and keep the blast radius small.

COBOL → Java Microservices Migration: Challenges & How We Solved Them

We're modernizing a mainframe COBOL system into Java Spring Boot microservices on Oracle, using an AI-assisted pipeline (Claude Opus) to analyze the legacy code and generate the new components. Here are the challenges that actually bite — and the engineering decisions that keep the migration safe.

The migration architecture at a glance

Here's the full picture — every component involved, from the legacy mainframe on the left, through the AI-assisted migration pipeline in the middle, to the new Java microservices on the right, with the equivalence-testing layer that de-risks the cutover underneath.

In one sentence: collect every legacy input → let the AI pipeline analyze & generate under strict guardrails → land it as domain-bounded Java microservices on Oracle → and only cut over once golden-master, parallel-run, and reconciliation prove the behaviour is identical.

Why this is hard (and why "just rewrite it" fails)

A COBOL system that's run a business for 30 years isn't just code — it's three decades of business rules, edge cases, and regulatory logic, much of it undocumented and written by people who've long since retired. Translating the syntax is the easy 20%. Preserving the exact behaviour is the hard 80%. Almost every failed migration we've seen failed on that 80%.

New to mainframes? A 60-second glossary

This post uses a few mainframe terms. If they're unfamiliar, here's the short version — you can read the rest without prior mainframe experience.

The challenges — and how we solve them

1. Hidden, undocumented business logic

Rules live inside the COBOL: level-88 conditions, REDEFINES, GO TO flow, fall-through PERFORM, and ON SIZE ERROR handlers. Documentation is stale.

How we solve it: we treat docs as a starting point, not truth, and verify behaviour against the real system (see the testing section). Our analysis step explicitly hunts for these constructs instead of assuming clean structured code.

2. Numeric precision — the silent killer

COBOL COMP-3 packed decimal with fixed-point arithmetic and explicit ROUNDED behaviour. If the generated Java uses double/float for money, the numbers will drift — and in finance, a cent of drift is a production incident.

How we solve it: every numeric PIC maps to BigDecimal with explicit scale and the exact RoundingMode. We replicate COBOL truncation/rounding precisely, and we verify it with reconciliation tests — never trust the model to "get math right" by default.

3. Avoiding the distributed monolith

COBOL has no domain boundaries — everything shares working storage. Convert one program to one microservice and you get a distributed monolith: all the coupling of the original, plus network latency and partial failure. Worse than what you started with.

How we solve it: Domain-Driven Design. We define service boundaries by business capability, not by screen or program, and migrate incrementally with the Strangler Fig pattern so old and new run side by side.

4. Transactions & consistency (CICS → microservices)

CICS (Customer Information Control System — the mainframe's transaction monitor) gave us ACID (atomic, consistent, isolated, durable) transactions across everything. Pseudo-conversational state and two-phase commits don't survive a move to distributed services.

How we solve it: keep a transaction inside one service wherever possible; use the Saga + Outbox patterns for the genuinely cross-service flows; design boundaries to minimize distributed transactions in the first place.

5. Data migration (VSAM/DB2/flat → Oracle)

EBCDIC→ASCII encoding, hierarchical/denormalized copybooks reshaped into normalized Oracle tables, six-digit date windowing.

How we solve it: ETL with field-level reconciliation reports (row counts, checksums, value diffs) and a dual-run period before cutover.

6. Batch & JCL — the part everyone forgets

Nightly batch jobs — orchestrated by JCL (Job Control Language, the mainframe's batch scripting) doing sort/merge and sequential processing — are a huge slice of mainframe value and don't map to request/response services.

How we solve it: rebuild it on Spring Batch, rethought as chunked/parallel/streaming — and test it the same way as the online flows.

7. Performance regression

Mainframe batch is brutally fast. Java + network hops + Oracle + chatty service calls (N+1 on the data gateway) can be slower.

How we solve it: performance-test on realistic data volumes early, and watch DB access patterns from day one rather than at the end.

Our AI-assisted pipeline — what works

We use Claude Opus to analyze COBOL programs and generate the Java components, but with hard guardrails. The things that make it reliable:

• Upfront dependency collection. We scan every CALL / PERFORM / COPY and refuse to start until all called programs, copybooks, and mapsets are present. Missing dependencies are the #1 cause of incomplete migrations — so we made it a blocking gate.
• 100% field-completeness validation. We extract every field reference number from the documentation and verify the generated code matches — catching the silent dropping of fields that AI is prone to.
• A single source of truth. Generated code may only use database field names from the field-mapping JSON — never invented ones. This is the key guardrail against the model hallucinating columns.
• Separation of client vs backend validators and registered enums, so the output is consistent across hundreds of programs instead of a different style each time.

The thing that actually de-risks the cutover: golden-master + parallel-run testing

There are no existing unit tests for a 30-year-old COBOL system. So how do you know the Java does the same thing? You don't reason about it — you measure it, with two complementary techniques.

Golden-master (characterization) testing

Capture a large set of real inputs and the mainframe's exact outputs as fixtures — production transaction logs are ideal because they cover the weird edge cases nobody documented. Replay those same inputs through the new Java service in CI and diff the outputs byte-for-byte (numbers, formatting, error codes, everything). Any mismatch is a regression to fix before launch.

• Pull fixtures from production data — synthetic test cases miss the 1-in-10,000 path that breaks in prod.
• Lock down non-determinism — freeze system date/time, sequence numbers, and user IDs so diffs reflect logic, not clocks.
• Diff at field granularity so a failure points to the exact field/rule, not "outputs differ."
• Run it in CI on every generated module — this becomes the regression suite the legacy system never had.

Parallel / shadow run

Before cutover, run both systems on live traffic: the mainframe serves users as normal, while the Java service processes the same requests in the background (shadow mode, zero user impact). Compare results continuously and alert on any divergence. Once divergence is near-zero for long enough, do a canary cutover — route a small slice of real users to Java, watch, then ramp to 100%, keeping the mainframe as instant rollback.

For batch and data, the equivalent is the dual-run + reconciliation described earlier: run both nightly, diff the outputs and the resulting Oracle state, and only retire the COBOL job once they match for a full cycle.

Takeaways

1. Migrate behaviour, not syntax — capture intent and prove equivalence.
2. Use BigDecimal with exact rounding for every number that matters.
3. Draw service boundaries by domain, not by program — or you'll build a distributed monolith.
4. AI accelerates the 80% of mechanical work — but keep hard guardrails (single source of truth, completeness gates) and humans + tests as the final authority.
5. Golden-master + parallel-run + reconciliation is what lets you cut over without holding your breath.

Event Fabric: Zero-Miss Corporate Action Notifications at Custodian Scale

In October 2008, Volkswagen briefly became the most valuable company on earth — not because of fundamentals, but because short-sellers hadn't accounted for a corporate action. Porsche's surprise disclosure that it controlled 74% of VW's shares triggered one of the most violent short squeezes in history. Corporate actions are low-frequency, high-stakes events. If your notification system misses even one, the consequences can be irreversible — missed rights deadlines, wrong tax withholding, compliance breaches. Here is how we built a system that doesn't miss any.

What is a Corporate Action, and Why Does Getting it Wrong Cost Millions?

A corporate action is any event initiated by a listed company that materially affects its securities or shareholders. Custodian banks (like Citi Global Custody, BNY Mellon, State Street) are legally obligated to notify every eligible client in time for them to act — or to act on their behalf when no instruction is received.

The industry standard notification format is SWIFT ISO 15022 MT564 (Corporate Action Notification) for legacy systems, now migrating to ISO 20022 SEEV.031. Citibank's Global Custody — one of the world's largest with assets under custody exceeding $25 trillion — pioneered Straight-Through Processing (STP) for corporate actions through their CAST (Corporate Action STP) initiative, eliminating manual touchpoints that historically caused notification failures. Our architecture draws from the same STP philosophy: every step automated, every failure surfaced, every message idempotent.

The System Landscape

Before touching a line of code, it helps to understand what each system owns and why the boundaries are drawn where they are.

Data Flow: A Dividend Hits the Wire

Let's trace a single event — Infosys declares a ₹21/share dividend with a record date of 2024-08-30 — from source to customer letterbox.

Step 1 — SMC → EBS: Daily Security Master Sync

Overnight, SMC pushes the full security master to EBS via batch. This includes the Infosys ISIN (INE009A01021), exchange (NSE/BSE), currency (INR), asset class (equity). EBS needs this to resolve positions correctly. Any ISIN that hasn't synced yet means EBS can't tell Aspen who holds it.

Step 2 — EMC → Aspen: Corporate Action Event via Kafka

EMC receives the dividend announcement (sourced from exchange feeds, DTCC equivalent in India is NSDL/CDSL depository notifications) and publishes to the corp-action-events Kafka topic. The message is keyed on isin + eventId to ensure all messages for the same event land on the same partition — preserving order.

Step 3 — Aspen: Entitlement Calculation

Aspen's Event Ingestion Service consumes the Kafka message. It then calls EBS (via Kafka reply or REST, depending on SLA) to fetch all customer positions in INE009A01021 as of the record date. This is not as simple as "who holds the stock today" — it must use the settled position snapshot at record date close, which means trades from T-1 may not yet be in the system.

Step 4 — Aspen → Apigee → Aggregator → TaxC + ED Services

For each customer, Aspen calls the Aggregator microservice through Apigee. The Aggregator orchestrates two parallel downstream calls: TaxC (to compute the withholding tax) and ED Services (if it's a Swiss-domiciled account, encode the account reference in the required format).

Step 5 — Aspen: PDF Generation & File Server Storage

The PDF is generated using a Thymeleaf/FreeMarker template engine running inside Aspen's PDF Generation Service. Each customer gets a personalised advice letter — their name, account number, the gross/net entitlement, tax breakdown, and payment date. The PDF is written to a shared object store (NFS/S3-compatible), and the server path is captured.

Step 6 — AGS: Fan-out to Delivery Channels

AGS consumes the ags-notifications Kafka topic. For each message, it reads the customer's delivery preferences from a config store and fans out accordingly. A customer with both electronic and paper preferences gets both — InView for immediate access and Print Vendor for the physical letter.

Problems We Hit in Production — and How We Solved Them

Problem 1 — EMC Duplicate Events (The "Double Advice" Incident)

EMC occasionally resends the same corporate action event — a network timeout causes a producer retry, or a manual re-announcement corrects minor data. In our first production run, 3% of customers received duplicate advice letters for the same dividend. Complaints started within hours.

Root cause: Aspen was not idempotent. It treated every Kafka message as a new event.

Fix: We introduced a deduplication table in PostgreSQL. Before processing any event, Aspen checks if (eventId, customerId) has already been processed successfully. We made the insert idempotent using an ON CONFLICT DO NOTHING clause, ensuring only the first message per pair triggers processing — even under parallel consumer restarts.

Problem 2 — The Race Between Record Date and Settlement (T+2 Hell)

Indian equity markets settle on a T+2 cycle. A customer buys Infosys shares on August 28 (two days before the record date of August 30). Their trade is confirmed on August 28 in EBS, but the shares don't settle into their account until August 30 — the record date itself, and often after market open. EMC fires the event at 9:00 AM on August 30. EBS queried at that moment shows the position as pending settlement, not settled. We excluded the customer. They complained — rightly — that they were entitled.

Fix: We introduced a settlement wait window. Aspen does not process events at the instant EMC fires them on record date. Instead, it schedules processing for 6:00 PM on record date, after which all same-day settlements are guaranteed to have cleared through the depository. EMC events arrive in a staging queue and are held until the processing window opens.

Problem 3 — TaxC Timeout for DTAA Edge Cases

For NRI customers with DTAA (Double Taxation Avoidance Agreement) treaties, TaxC needs to verify the customer's tax residency certificate (TRC) stored in a separate document management system. This lookup occasionally took 8–12 seconds — far beyond our 3-second SLA for the Aggregator call. The Aggregator was timing out, causing Aspen to fall back to the default 20% rate — which for some customers was higher than their treaty rate, resulting in over-deduction.

Fix: We split the processing path. Customers with DTAA eligibility flags are routed to an async enrichment queue. Aspen generates a preliminary advice with the default rate and marks the record as PENDING_DTAA_ENRICHMENT. A separate job polls TaxC for the resolved rate, regenerates the PDF if the rate changed, and marks delivery complete only then. We added a 48-hour SLA on the async path with an alert if it breaches.

Problem 4 — Swiss Account Encoding Failures (ED Services Edge Case)

ED Services translates account identifiers between IBAN format (used by Swiss banks — e.g., CH56 0483 5012 3456 7800 9) and the internal custodian reference format. This worked fine for 99.8% of accounts. The 0.2% that failed had beneficial owner names containing umlauts (ä, ö, ü, ß) — stored as UTF-8 in our system but expected as ISO-8859-1 by the downstream encoding engine. Encoding failures caused those accounts to be skipped entirely with no error surfaced to operations.

Fix: We added a character encoding normalisation step in the Aggregator before calling ED Services. We also added a dead-letter topic (ed-services-dlq) so encoding failures are visible in the ops dashboard rather than silently swallowed.

Problem 5 — Kafka Consumer Lag During Dividend Season

The Indian market pays the bulk of dividends in Q1 (April–June) and Q4 (January–March). During Infosys, TCS, and Wipro dividend week — all three announcing within 48 hours of each other — EMC was producing 40,000 events/second. Aspen's consumer group had 12 partitions and 12 consumers. The processing lag hit 4 hours, which meant some customers were receiving advice after payment date — legally problematic.

Fix: We introduced horizontal scaling triggers via Kubernetes HPA (Horizontal Pod Autoscaler) keyed to consumer lag, monitored through Kafka's consumer_lag metric exposed to Prometheus. When lag exceeded 10,000 messages, HPA scaled Aspen's PDF generation pods from 12 to 48. We also moved PDF generation out of the main processing loop into a separate worker pool with its own autoscaling, so the entitlement calculation pipeline wasn't blocked by slow PDF rendering.

Problem 6 — AGS Thundering Herd on Large Dividends

A large Nifty 50 company paying dividend has ~500,000 eligible custodian customers. When Aspen finishes processing and publishes 500,000 messages to ags-notifications within 20 minutes, AGS received them all in a burst. AGS then fan-out called InView and Print Vendor simultaneously for each one. InView's REST endpoint was rated at 5,000 requests/minute. The result: 495,000 HTTP 429s (Too Many Requests), a full retry storm, and InView going down for 20 minutes.

Fix: We introduced a rate-limited work queue between the Kafka consumer and the delivery clients. AGS now pulls from Kafka at its own pace and uses a token bucket (implemented with Guava's RateLimiter) per delivery channel. InView is capped at 4,000 req/min (headroom below the 5,000 limit), Print Vendor at 1,000 req/min. We also staggered delivery: electronic first (time-sensitive), print batch dispatched in off-peak hours (11 PM).

Problem 7 — Exactly-Once Delivery and the Advice Letter Already Sent

Kafka guarantees at-least-once delivery by default. A consumer rebalance mid-batch caused AGS to reprocess 1,200 messages it had already sent to InView. Customers received their advice letter twice — not a compliance breach, but a significant customer experience failure and an operational support spike.

Fix: We enabled Kafka's transactional producer in Aspen and idempotent consumer pattern in AGS. AGS writes a delivery receipt to its own PostgreSQL table in the same transaction as the Kafka offset commit. If the consumer restarts and sees the same message again, the delivery receipt lookup short-circuits before the InView call is made.

Problem 8 — Voluntary Corporate Action Cut-Off Miss

For a rights issue, customers typically have 15–21 days from notification to instruct their custodian. We had one incident where a batch of 8,000 customers received their InView notification on Day 13 of a 15-day window — two days before cut-off. Many missed the deadline. Investigation found a Kafka consumer group rebalance had paused processing for 38 hours.

Fix: We added an SLA breach detector. For voluntary corporate actions, Aspen stamps an urgentDeliveryBy timestamp = recordDate minus 5 business days. AGS checks this timestamp and moves urgent events to a priority Kafka topic (ags-notifications-urgent) that bypasses the rate limiter and uses dedicated consumer threads.

Architecture Decisions We'd Make Differently

• Outbox pattern from day one. We added the transactional outbox (writing to DB + Kafka in one local transaction) six months after go-live, after the exactly-once incident. It should have been in the initial design. The Debezium CDC (Change Data Capture) pattern makes this clean — write to DB, let Debezium publish to Kafka from the WAL log.
• Separate the PDF storage path from the processing path earlier. Having Aspen write PDFs to a file server and then Kafka-notify AGS with the path is correct — but we initially had Aspen embed the base64-encoded PDF in the Kafka message. At 200 KB average PDF size, a 500K-customer event produced 100 GB of Kafka data in a single topic. Kafka is not a file store. We refactored this in sprint 3, but it cost us two weeks.
• Event schema registry from day one. We used plain JSON Kafka messages initially. When EMC added two new fields to the corporate action event payload, three downstream consumers broke in prod. Had we used Confluent Schema Registry with Avro schemas, the compatibility check would have caught the breaking change in CI.
• Apigee circuit breaker configuration needs tuning per downstream. The default Apigee circuit breaker settings (50% failure rate over 10 seconds) were too aggressive for ED Services, which has naturally higher latency for Swiss accounts. It was tripping the circuit and sending Swiss account requests to the fallback path when ED Services was healthy but slow. We tuned the circuit breaker to 70% failure rate over 30 seconds for ED Services specifically.

Key Learnings

1. Idempotency is not optional in financial systems. Every system in the chain — Aspen, AGS, InView client — must be idempotent. Budget for it upfront. It touches your DB schema, your Kafka config, and your HTTP retry strategy.
2. Record date ≠ processing time. Corporate action timing is deeply intertwined with settlement cycles. Build the settlement wait window into your design, not as a hotfix.
3. Tax logic is a first-class domain. The temptation to inline TaxC's rules into Aspen is real. Resist it. Tax rules change every budget season. A dedicated service with its own release cycle is what lets you update 194/195 TDS rules without redeploying Aspen.
4. Dead-letter queues are your audit trail. Every integration point (ED Services, InView, Print Vendor) should have a DLQ. Ops should have a dashboard showing DLQ depth. Silence is not success — it might be silent failure.
5. Kafka is not a database. Do not embed large payloads in messages. Kafka messages should carry references (PDF path, event ID), not the data itself. The file store is the single source of truth for the PDF; Kafka is just the notification.
6. The ISO 20022 migration is coming for everyone. The Indian securities market (NSDL, CDSL) is progressively aligning to ISO 20022 message formats (SEEV.031 for notifications, SEEV.033 for instructions). If you're building corporate action pipelines today, design your internal event schema to map cleanly to SEEV.031 fields — the migration will be far less painful.
7. Voluntary and mandatory events need different pipelines. Mandatory events (dividends, bonus, splits) need fast, high-throughput processing. Voluntary events (rights, buybacks, proxy votes) need SLA-tracked, deadline-aware pipelines with customer instruction collection. Running them through the same queue means mandatory events can delay voluntary ones during peak load — exactly when you can't afford delays on voluntary cut-off dates.

Industry Reference Points

Putting an LLM Agent in Front of a Live CRM — Without Letting It Invent Prices

The problem with chatbots on agency websites

Most website chatbots are a search box wearing a costume. They paraphrase your FAQ page, get confidently wrong about anything commercial, and hand off to a form the moment the conversation gets useful.

The failure mode that actually matters is pricing. A visitor asks "what does a business website cost?" and a naive assistant answers from whatever it absorbed during training. It sounds authoritative. It is not your price. You now have a prospect anchored on a number you never quoted, and a conversation that starts with a correction.

So the design goal was narrower than "build a chatbot": the model should be structurally incapable of stating a price we do not publish, and it should be able to complete real transactions — create a CRM lead, check calendar availability, book a consultation — rather than deflecting to a form.

Architecture

The assistant is not a separate service. It runs inside the existing Spring Boot monolith and calls the same repositories and mail sender the website forms already use. That was deliberate: a lead created by the assistant should be indistinguishable from a lead created by the contact form, because the sales team should not have to care which one produced it.

Visitor (browser widget)
          │  POST /api/chat/stream       (SSE)
          ▼
  ChatController ── rate limiter ── transcript writer
          │
          ▼
  ChatService ──► Claude API (tool runner)
          │              │
          │              ├── get_pricing        → PricingCatalog (in-process)
          │              ├── create_lead        → LeadRepository + JavaMailSender
          │              ├── check_availability → BookingRepository
          │              └── book_consultation  → BookingService
          ▼
  ChatSession / ChatMessage  (Postgres)
  

Everything below the tool boundary is code that already existed and was already trusted. The model does not write SQL, does not compose emails, and does not decide what a package costs. It decides which tool to call and with what arguments — and every tool validates its own inputs as if the caller were hostile, because functionally it is.

The pricing tool: making hallucination structurally impossible

The single most valuable design decision was refusing to put prices in the system prompt.

Prices in a prompt are just tokens. The model will happily interpolate between them, round them, or blend them with something it half-remembers about Indian web development rates. Instead, the catalogue lives in a plain Java component, and the prompt contains an instruction with teeth:

Call the GetPricing tool before you state any price, package name, or delivery timeline. Never quote a figure from memory and never estimate one yourself.

@JsonClassDescription("""
      Look up the real published pricing. Call this before stating any price,
      package name, or timeline — never quote figures from memory.""")
  public static class GetPricing implements Supplier<String> {
  
      @JsonPropertyDescription("One of: website, design, app, software, marketplace.")
      public String service;
  
      @Override public String get() {
          return pricing.describe(normalise(service));
      }
  }

The tool returns formatted text straight from the same source of truth the public pricing calculator renders from. When we tested it, the assistant quoted "Rs 60,000 to Rs 1.2 lakh" for a business website — an exact match for the catalogue entry, because it had no other option.

Tools that write: the rules change

Read-only tools are easy. The moment a tool creates a database row or sends an email, you are letting a probabilistic system take an irreversible action, and the design has to reflect that.

Validate as if the model is adversarial

Not because it is malicious — because it is helpful, and helpfulness under pressure looks like inventing a plausible email address to complete a task. Every write tool re-validates:

if (name == null || name.isBlank()
          || email == null || !EMAIL.matcher(email.trim()).matches()) {
      return "FAILED: a valid name and email address are required. "
           + "Ask the visitor for the missing detail before trying again.";
  }

Note the shape of the failure. It is not an exception and not a silent no-op — it is a sentence addressed to the model, telling it what to do next. The prompt has a matching rule: "If a tool returns a line starting with FAILED, tell the visitor plainly what is missing and ask for it. Do not retry with invented values." Failure messages are part of your prompt surface, and writing them as instructions rather than error codes measurably improves recovery.

Re-check preconditions at write time

The booking flow has a classic time-of-check/time-of-use gap. The model calls check_availability, tells the visitor the morning slot is free, and the visitor spends four minutes finding their phone number. In that window the slot can fill.

So book_consultation re-counts capacity immediately before insert, and refuses if the slot filled since the check:

long used = bookings.findByDate(day).stream()
          .filter(b -> slot.equals(b.getTimeSlot()))
          .filter(b -> !"CANCELLED".equals(b.getStatus()))
          .count();
  
  if (used >= CAPACITY_PER_SLOT) {
      return "FAILED: that slot has just been taken. "
           + "Offer the visitor a different slot or day.";
  }

We tested this by filling the slot to capacity behind the model's back and then pushing hard — "Confirmed, go ahead and book it." It refused and offered alternatives. Zero rows written. That test matters more than any number of happy-path conversations.

Linking a tool call back to its conversation

The SDK instantiates tool classes by deserialising the model's arguments onto their fields, so they cannot take constructor arguments — there is no clean way to hand a tool the current session. We used a ThreadLocal set around the tool-runner call:

ChatContext.set(sessionId);
  try {
      for (BetaMessage msg : client.beta().messages().toolRunner(params)) { … }
  } finally {
      ChatContext.clear();
  }

Not elegant, but correct for a blocking, one-conversation-per-thread model, and it lets create_lead attach the resulting lead to the transcript that produced it.

Streaming through a tool loop

This was the part we expected to be hard, and it was — just not for the reason we predicted.

Token streaming is straightforward when the model simply answers. It gets awkward when the model might pause mid-turn to call a tool, because a naive implementation has to reconstruct tool-call arguments from input_json_delta fragments, execute the tool, rebuild the assistant message, and resume — all while keeping a socket open to the browser.

We were preparing to hand-roll that when a look through the SDK jar turned up BetaToolRunner.streaming(), which returns one stream per turn of the agent loop and executes tools between turns:

for (var streamResponse : runner.streaming()) {
      try (streamResponse) {
          streamResponse.stream().forEach(event -> {
              event.contentBlockStart().ifPresent(start ->
                  start.contentBlock().toolUse().ifPresent(t -> sink.tool(t.name())));
  
              event.contentBlockDelta().ifPresent(delta ->
                  delta.delta().text().ifPresent(t -> sink.text(t.text())));
          });
      }
  }

The narration mismatch

A subtle bug fell out of this. On a tool turn the model often writes something first — "Let me check our current pricing" — then calls the tool, then answers on the next turn. Our first implementation kept only the final turn's text as the reply.

The visitor saw both, because both were streamed. The stored transcript contained only the second. So the conversation history we replayed to the model, and the record the sales team read, both disagreed with what the customer actually saw.

The fix keeps every turn's text, joined with a paragraph break — and crucially pushes that separator to the client too, so the rendered text and the stored text are byte-identical:

// Paragraph break between a pre-tool narration and the answer that
  // follows, so the stream matches the stored text.
  if (turnText.isEmpty() && !answer.isEmpty()) sink.text("\n\n");

The fallback that would have doubled every broken answer

We kept a non-streaming endpoint as a fallback for browsers or proxies that mishandle the stream. The first version guarded it like this:

.catch(function () {
      if (busy) plainRequest();   // ← wrong
  });

The comment above it claimed "only fall back if nothing was rendered." The code did not do that. busy stays true until the answer completes, so a stream that died halfway would retry and append a second answer underneath the half-rendered first one. Tracking what was actually painted fixes it:

if (!busy) return;                    // already resolved
  if (streamRendered) {                 // half an answer is on screen
      fail('That reply was cut off. Please ask again.');
  } else {
      plainRequest();                   // nothing shown yet — safe to retry
  }

Worth stating plainly: the comment was right and the code was wrong, which is the most dangerous combination in a review. A reviewer reading the comment would have approved it.

The bug that took down every script on the page

The assistant emits light Markdown, so the widget renders bold and links — escaping first, then re-introducing only <strong> and https anchors, so a hostile reply cannot inject markup.

The link regex contained this fragment:

/\[([^\]\n]+)\]\((https:\/\/[^\s)]+)\)/g

Which contains the literal two-character sequence [( — and this page is rendered by Thymeleaf, where [(…)] is inline-expression syntax.

Thymeleaf tried to parse it, failed, and truncated the entire HTTP response mid-stream. Not the widget — the whole page. Every script after that point never loaded. The browser reported ERR_INCOMPLETE_CHUNKED_ENCODING and an unrelated-looking SyntaxError: Invalid regular expression, and the page simply stopped at safe.replace(/\.

The fix is one attribute, with a comment so nobody "tidies" it away:

<!-- th:inline="none" is required: the markdown regex below contains the
       literal sequence "[(", which Thymeleaf otherwise parses as an inline
       expression and then fails on, truncating the whole response. -->
  <script th:inline="none">

Cost control

Two decisions kept running cost negligible:

Prompt caching. The system prompt and catalogue instructions are static and substantial. Marking them cacheable means subsequent turns in a conversation re-read them at a fraction of full input cost.

Lazy qualification. We extract a structured qualification record — service, budget, timeline, urgency, qualified yes/no — from each transcript using structured outputs. Running that on every turn would roughly double spend for conversations that never convert. Instead it runs when a lead or booking is created (the moment it is worth money) or on demand from the admin screen. Sessions nobody looks at cost nothing extra.

Design decisions

The availability trap

One decision we got wrong first time is worth the warning.

Adding the assistant meant removing the floating WhatsApp button — two floating buttons is clutter. The widget hid itself when the assistant was unavailable, which seemed tidy: don't invite a conversation you can't have.

Then we deployed before setting the API key in production. The assistant reported unavailable, the launcher hid itself, and the site had no floating contact button at all. We had coupled a proven conversion path to the availability of a brand-new dependency.

The launcher is now always visible. When the assistant is offline the panel becomes a WhatsApp hand-off with a grey status dot instead of green. Worst case is "no AI", never "no way to reach us".

Takeaways

1. Constrain, don't instruct. "Never invent prices" in a prompt is a wish. A tool that returns the only prices that exist is a guarantee.
2. Write tools are a different risk class. Validate inputs, re-check preconditions at commit time, and test the refusal path harder than the success path.
3. Read the SDK before hand-rolling. The streaming tool runner already existed; we nearly built and owned a worse version of it.
4. What the user saw is the source of truth. If your transcript disagrees with the rendered conversation, both your model context and your sales team are working from fiction.
5. Never let a new dependency become a single point of failure for an old, working path.

We Killed Our Handwriting AI After Five Rounds. Here's the Number That Ended It.

Most write-ups of AI projects are about the ones that worked. This is about one that didn't, because the reason it failed is more useful than another success story — and because we very nearly shipped it anyway.

What we were building

Two related products. Invoice AI: photograph a supplier invoice, extract the line items, validate the totals, present them for review. Boutique Manager: the same idea aimed at handwritten order slips for a boutique client of ours.

The pitch writes itself. A shop owner photographs the day's bills instead of typing them. Vision models read documents well. Everyone has seen the demos.

We set an accuracy gate before writing any product code: 95% field-level accuracy on real documents. Below that, a human has to check every field, and if a human checks every field the product saves nobody any time.

The spike

We ran fourteen real handwritten bills from the boutique through five extraction strategies:

1. Whole-page extraction — the entire photo, one prompt, free-form JSON out.
2. Fitted schema — the same images with a strict output schema constraining every field's type and shape.
3. Zone cropping — cut the bill into regions (header, customer block, line items, totals) and extract each separately.
4. Hybrid routing — printed regions one way, handwritten regions another.
5. A second, better-lit batch — because by then we suspected the inputs.

Printed content — headers, pre-printed field labels, the shop's own letterhead — came out fine throughout. That is the part everyone demos.

The handwritten fields never came close. Dates, customer names and mobile numbers failed persistently across all five rounds. More cropping did not fix it. A tighter schema did not fix it. Better lighting did not fix it.

The number that actually ended it

Low accuracy alone would not have killed the project. You can design around a model that says "I can't read this" — flag it, route it to a human, extract the other eleven fields, still save time.

What killed it was how it was wrong.

Sit with that. Not a refusal. Not a low-confidence flag. Not a partial read with the illegible digits missing. Two well-formed, ten-digit Indian mobile numbers, neither of which appeared on the bill, both asserted at a confidence level that any sane pipeline would auto-approve.

This breaks the standard mitigation. The usual architecture for imperfect extraction is a confidence threshold: auto-accept above it, queue for human review below. That design assumes confidence correlates with correctness. Here it did not correlate — and worse, it was anti-correlated in the specific cases that mattered, because a clean fabrication looks more like a valid phone number than a genuine partial read does.

A review step cannot catch this either, at least not one a human will actually perform. Reviewing a blank field takes a second — your eye goes straight to the gap. Reviewing a plausible ten-digit number means comparing it digit by digit against a smudged photograph, for every field, on every bill. Nobody sustains that past week one. The review step degrades into rubber-stamping, and now you have wrong customer phone numbers in your database with an audit trail saying a human approved them.

Why we did not "just prompt better"

We tried the obvious things: explicit instructions to return null rather than guess, schema constraints that permitted an empty value, prompts that named the failure mode directly.

They helped at the margins. They did not change the shape of the problem, and in hindsight it is clear why. The model was not making a reasoning error we could correct with better framing. It was doing exactly what a next-token predictor does with an ambiguous visual input: producing the most plausible continuation. A ten-digit number in a field labelled "Mobile" is overwhelmingly plausible. The model had no mechanism to distinguish "I read this" from "this is what such a field usually contains", and the confidence score reflected the plausibility of the output, not the legibility of the input.

That is not a prompt bug. That is the tool being asked to do something outside what it can structurally guarantee.

The mistake we made in the evaluation itself

This part is more embarrassing than the failure, and more useful.

The product was specified for printed GST supplier invoices. Every photograph we actually received and tested was a handwritten sales slip — the opposite side of the transaction, and a far harder document class. On top of that, every image had been sent through a messaging app and arrived compressed to about 1.9 megapixels.

So we ran five rounds of increasingly sophisticated extraction strategies against the wrong document type at degraded resolution, and drew a conclusion about the product.

The conclusion to stop was still correct — for the boutique use case, which was genuinely handwritten. But we never tested the thing we said we were building, and we would not have noticed if the numbers had come out slightly better. A marginal pass on the wrong corpus would have shipped a product that failed differently in production.

What we kept

Killing a project is not the same as wasting the work. Three things survived:

A reusable evaluation harness. The scaffolding for running a corpus through multiple strategies and scoring field-level accuracy outlived the project and is the reason the next AI feature could be assessed in days rather than weeks.

A written record of Claude API findings — structured-output limits, caching minimums, and the central one: model confidence cannot be used to route escalation on this class of task. That single line has since prevented us proposing at least one architecture that would have had the same flaw.

The one product idea worth keeping. During scoping we realised the valuable thing was not reading the slip — it was storing garment measurements against the customer rather than against the order, so a returning customer's sizes are already known. That is a data-model insight, and it needed no AI at all. If a boutique system is ever built, capture will be typed, not photographed.

Design decisions

How to run this test on your own AI feature

If you are evaluating an extraction or classification feature, these four checks will tell you more than an accuracy percentage:

1. Feed the same input twice. Different answers on identical input means the model is generating, not reading. Do this before anything else — it is one API call and it is decisive.
2. Plot confidence against correctness. If they do not correlate, every threshold-based routing design you were planning is already dead. Find this out on day two, not in production.
3. Count fabrications separately from misses. A missed field costs a few seconds. A fabricated field costs trust, and it is invisible until a customer complains.
4. Check your corpus is the corpus. Ours was not, and five rounds of good engineering went into answering the wrong question.

Takeaways

1. Accuracy is the wrong headline metric. The distribution of failures matters more than the rate. A 90% model that fails loudly beats a 94% model that fails silently.
2. Confidence scores are not reliability estimates. On ambiguous visual input, confidence tracks how plausible the output looks — not how well the input was read.
3. "Add a human review step" is not a plan unless the errors are the kind a human will actually catch on the hundredth document, tired, at 7pm.
4. Cheap kills are a capability. Five rounds and fourteen documents cost days. Shipping it would have cost a client's data integrity and our credibility.
5. Knowing when not to use AI is engineering judgement, and it is worth more to a client than another integration.

From HTTP JSON to a Raw TCP Device Gateway: Building GPS Tracking That Survives the Field

Vehicle tracking looks trivial until you meet an actual GPS tracker. The prototype is an afternoon: accept a lat/long over HTTP, write it to Postgres, push it to a map. Then you plug in hardware and discover the device has never heard of HTTP, has no TLS stack, and communicates by opening a TCP socket and emitting bytes in a proprietary format for as long as it has signal.

This is how we got from one to the other, and which decisions turned out to be load-bearing.

Three stages, deliberately

We built the system in three passes, each replacing exactly one assumption. That sequencing mattered more than any individual design choice, because it kept a working system at every step.

Notice what never changed after stage 1: the persistence layer, the query API, and the entire React dashboard. That is not luck — it is the payoff from one boundary described below.

The boundary that made the rewrite survivable

The most valuable object in the system is a plain DTO called NormalizedLocation.

Dummy GPS Tracker (TCP, $7DS,…*CS)
          │
          ▼
     Device Gateway      ← sockets, framing, connection limits
          │
     Session Registry    ← auth state, one session per device
          │
     Protocol Adapter    ← parse packets, build responses
          │
     Device Registry     ← credentials → vehicle mapping
          │
          ▼
    ┌─────────────────┐
    │ NormalizedLocation │  ◄── the boundary
    └─────────────────┘
          │
     TrackingService     ← unchanged since stage 1
          │
     ┌────┴────┐
     ▼         ▼
  Postgres   SSE ──► React + Leaflet dashboard
  

Everything above that line is device- and vendor-specific. Everything below it knows nothing about how a fix arrived.

That is why swapping HTTP JSON for framed TCP touched no persistence, no query API and no UI code. TrackingService receives the same object it always did. Adding a real vendor — Teltonika, Concox, whatever the client's fleet actually runs — means writing one adapter that terminates at the same DTO, not threading a new protocol through the application.

Framing: the class that decides whether you survive production

This is the part that separates a demo from a deployment, so it deserves the detail.

TCP is a byte stream, not a message stream. The protocol guarantees ordered, reliable bytes. It guarantees nothing whatsoever about how those bytes are grouped into read() calls. A single read may return half a packet, three packets, or two and a half.

The naive gateway does this:

// Wrong. Works perfectly on your desk.
  int n = in.read(buf);
  handlePacket(new String(buf, 0, n));

On a LAN, with one device sending one small packet every ten seconds, each write arrives as exactly one read. The demo is flawless. Then you deploy to a fleet on mobile networks, and Nagle's algorithm, network buffering and retransmission start coalescing and splitting writes. Now you drop packets and corrupt others.

The failure characteristics are what make this genuinely dangerous:

• It is intermittent — most packets still parse fine.
• It is load-dependent — it appears when the fleet grows, not when you test.
• It looks like hardware flakiness — the natural conclusion is "the tracker is dropping fixes", and you go debugging the wrong system.

The fix is a framer that buffers bytes and emits only complete messages, delimited by newline:

3 packets in one TCP write    → 3 ACKs
  1 packet split over 3 writes  → 1 ACK, only after the final fragment
  8 KB with no delimiter        → connection dropped

That third line is not an edge case, it is a denial-of-service defence. A peer that opens a socket and never sends a delimiter would otherwise grow your buffer until the process dies. Capping an unterminated run at max-packet-size costs four lines and removes an entire class of outage.

Authenticating the connection, not the packet

Stage 2 had a hole we shipped knowingly and closed in stage 3, and it is a good illustration of how identity works differently on a socket.

In stage 2, any packet naming a registered device was honoured. Since the device id travels in the packet, an arbitrary TCP client could write location fixes for somebody else's vehicle. There is no cookie, no bearer token, no TLS client certificate — just bytes on a socket claiming to be a device.

Stage 3 scopes authentication to the connection. Every socket owns a session:

CONNECTED ──AUTH ok──► AUTHENTICATED ──displaced/shutdown──► CLOSING ──► CLOSED

carrying connectionId, remoteAddress, state, deviceId, connectedAt, authenticatedAt, lastPacketAt and a packet count.

Before authentication, only an AUTH packet is honoured. A connection that has not proved credentials cannot submit location or heartbeat data at all. Credentials are checked once against the registry — device id, IMEI, protocol and administrative status — and the resulting identity is bound to the socket rather than re-asserted per packet.

The other session rule worth stealing: duplicate connections resolve newest-wins. A tracker that loses signal in a tunnel often reconnects before the old socket's keepalive expires, leaving two live connections claiming the same device. Accepting both means interleaved fixes and a vehicle that teleports on the map. The new connection displaces the old one, which is closed.

Two timeouts that sound identical and are not

This one cost us a confusing debugging session and is worth naming explicitly.

A device can show OFFLINE on the dashboard while its TCP connection is perfectly open, and vice versa. Conflating them produces bugs that look supernatural — a vehicle greying out while data flows, or showing live minutes after the socket died. Keep them separately configured and separately named, and write the distinction down where the next engineer will find it.

Virtual threads make the blocking model viable again

The gateway uses one virtual thread per device connection. A blocking read() then costs a stack rather than an OS thread.

This is a genuinely material simplification. The classic answer to thousands of mostly-idle sockets is non-blocking IO — Netty, an event loop, callbacks, and a codebase where the connection lifecycle is scattered across handlers. With virtual threads you write the obvious sequential code: read, frame, parse, authenticate, respond. Each connection's logic reads top to bottom, and the platform handles the scheduling.

For a fleet gateway — high connection count, low per-connection throughput, long idle periods — this is close to an ideal fit.

Everything is configurable, because field conditions are not negotiable

tracking:
    device-gateway:
      enabled: true
      tcp-port: 9000              # 0 binds an ephemeral port; the tests rely on this
      idle-timeout-seconds: 60
      max-packet-size: 4096
      max-connections: 100
      last-seen-flush-seconds: 30
    gps:
      future-tolerance-seconds: 120

Two details worth copying. Port 0 binds an ephemeral port, which is what lets the integration tests start a real gateway on a free port and exercise genuine sockets rather than mocks — the framing behaviour above is only meaningfully testable against real TCP. And every property has an env-var override, because the tolerances you need for a warehouse yard and a long-haul fleet on patchy rural coverage are not the same numbers, and neither is worth a redeploy.

future-tolerance-seconds exists because trackers lie about time. A device with a drifting RTC or a cold GPS fix will report timestamps minutes or years in the future. Accepting them silently corrupts every trip calculation downstream, so fixes beyond tolerance are refused at the boundary rather than cleaned up later.

Design decisions

What this deliberately is not

Scope discipline is part of the design. The POC is a device gateway and a live map. It is not a fleet management product — no trips, no geofences, no alerts, no driver behaviour scoring, no historical replay. Those are real features and they belong above TrackingService, which is precisely why the boundary was worth establishing before any of them existed.

The honest limitation: the wire protocol is our own $7DS,…*CS format, exercised by a dummy tracker we wrote. Real vendor protocols are messier — binary framing, checksums that disagree with their own documentation, undocumented heartbeat variants. The architecture anticipates them through the adapter interface, but adapter number one being cooperative is not the same as having integrated a real fleet, and we would not claim otherwise.

Takeaways

1. Find your normalisation boundary before you integrate hardware. It is what makes the second vendor cost a week instead of a rewrite.
2. Framing is not a detail. If you take one thing from this: never assume one read() is one message. It works on your desk and fails in the field, intermittently, looking like someone else's bug.
3. Identity belongs to the connection. On a raw socket there is no session cookie — if you do not bind identity to the connection, the packet gets to claim whatever it likes.
4. Name your timeouts precisely. Resource reclamation and user-facing status are different concerns that sound like the same word.
5. Virtual threads change the calculus. Thread-per-connection with blocking IO is viable again, and the resulting code is dramatically easier to reason about than an event loop.

58 Tools, 18 Screens: Building a Flutter App Where Tools Are Data, Not Code

The brief was an all-in-one utility app targeting 500+ tools. The obvious implementation — one screen per tool — collapses immediately under its own arithmetic: 500 screens, 500 sets of validation, 500 test suites, and a store release every time you fix a typo in a calculator.

So the architecture had to answer one question before any code was written: what is a tool, such that the 500th costs one file?

Tools are data, not screens

A tool is a registry row plus a runner. The row is a ToolSpec served from the backend; the runner is a Dart class implementing a single method:

FutureOr<ToolResult> run(ToolInput input);

alongside a declared List<ToolField> describing its inputs. One universal ToolShellScreen renders those fields, collects values, calls run, and renders the result.

Eighteen screens serve fifty-eight tools, and would serve five hundred. The screens that exist are the shell, the studio hubs, search, favourites, profile — structural surfaces, not per-tool ones. Adding a percentage calculator is one Dart file and one registry row. No new navigation, no new layout, no new tests beyond the transform's own.

Four execution tiers, chosen at runtime

The genuinely useful decision was making execution strategy a registry field rather than a code-level choice. Every tool declares a tier:

Because tier lives in the registry, a tool can change execution strategy without an app release. Ship a tool as web to get it into users' hands, port it to native later, flip the field — installed apps pick up the change on next catalogue fetch. That single indirection converts a store release into a database update.

Where the engine refuses to go

The most instructive decision was the work we declined to make the engine do.

run(input) → result expresses transforms: inputs in, result out. It cannot express anything continuous, ambient or device-driven. Four catalogue entries hit that wall:

• Stopwatch and countdown timer — continuous, must run in background
• Flashlight — device control with no output at all
• Document scanner — camera-driven, multi-step, stateful

The tempting move is widening ToolRunner until these fit. We left them unimplemented instead, and eventually retired three from the catalogue.

The reasoning: the engine's entire value is that tool 500 costs one file, and that holds because it refuses work it cannot model. Forcing a stopwatch through a request/response interface either fakes the abstraction or warps it for the other fifty-seven tools. The correct answer to "this doesn't fit the engine" is a second shape — a bespoke screen — not a wider interface.

There is a real distinction worth preserving here. When nested output turned out to be a genuine gap, we did extend the result model (ResultSection, ResultItem.code). Extending the result model to carry real payloads is correct; extending the execution model to fake a different interaction is not.

Why the mobile API is a separate module

The obvious shortcut was extending the existing website backend. We built a separate tools-api service instead.

The website is a multi-tenant SSR CMS with session-cookie auth and a tenant interceptor resolving brands from the Host header. The mobile API needs stateless JWT, Redis, and independent scaling — every one of those in tension with the CMS's request model. Bolting mobile onto it means either contorting the tenancy model or running two auth schemes in one app.

The existing endpoints are consumed through a LegacyToolsClient facade and never modified. Sixteen of the twenty-one web tools were already 100% client-side and ported straight to the native tier; five had working backend endpoints already.

One decision from that module that repaid itself: guest is a real user row. Not a null, not a special case — a row. That removes every "if user is null" branch in the codebase, and signing in becomes an upgrade of an existing record rather than a migration between two states.

Then you meet Google Play

The engineering was the easy half. What follows is the part nobody documents, and it is the reason a finished app can sit unreleased for a month.

The gate nobody warns you about

To publish from a personal developer account, Google requires 12 testers opted in continuously for 14 days before you can even apply for production access.

Every word there is load-bearing:

• Opted in, not signed up. A tester who joins the mailing list but never accepts the invite and installs does not count.
• Continuously. The 14 days are unbroken. Someone uninstalling on day 9 can reset your clock.
• Closed testing only. Internal testing days do not count. This is the expensive misconception — you can run internal testing for a month and have made zero progress toward the gate.

Add identity verification (a few days, and it needs a physical Android device for the device verification step) and review (about a week), and a finished app is roughly 3–4 weeks from its first upload to production. Plan the launch backwards from that, not forwards from "code complete".

Firebase decides your privacy disclosures for you

We declared Advertising ID: Yes, purpose Analytics. That was not a product choice. Firebase Analytics puts com.google.android.gms.permission.AD_ID in the merged manifest, and Play detects it during review. Declaring "no advertising ID" while shipping that permission is a rejection.

Declarations are judgement calls — write down the reasoning

Three that required actual thought:

Files and docs: Collected, required, not ephemeral. The PDF suite uploads to S3. "Deleted on completion" is not the same as Google's in-memory-only definition of ephemeral, so claiming ephemeral would have been wrong.

Health: not collected. BMI is 1 of 58 tools and is marked sensitive, so its input never leaves the device. That is why the app is not a health app and why Data Safety says health data is not collected — a defensible position, but only because of an architectural choice made earlier.

Financial features: none. Calculators are not financial services.

We keep these justifications in writing because a reviewer may query them, and the reasoning is not re-derivable from the code alone. "Why did you say health data isn't collected?" has a good answer, but only if someone recorded it.

Three Flutter and Android gotchas paid for once

Your TLD can break your package name. in is a Kotlin keyword, so the .in domain forces backticks in the Kotlin package declaration. Discovered at build failure, not at design time.

Never wildcard an assets directory that also holds source artwork. Declaring assets/branding/ shipped 3 MB of full-resolution originals into the APK. List asset files explicitly, or keep sources outside the asset tree entirely.

Test the release build, not the debug build. On a memory-constrained machine, an emulator plus a debug Flutter build exhausts RAM and Android's low-memory killer starts taking down Play Services and the app. Debug cold start is ~9 s versus ~1.3 s for release — a "hung" splash screen is usually just JIT warm-up, and chasing it as a bug wastes an afternoon. The release APK came in at ~21 MB against a 25 MB budget.

Design decisions

Freeze the release, not the branch

One process decision worth stealing. When the build went to testing, we tagged it v0.1.0 and stopped touching the version in pubspec.yaml while that build was live with testers.

The reason is narrow and practical: a tester bug report arriving three weeks later can be reproduced with git checkout v0.1.0 rather than archaeology across a fortnight of commits guessing which ones were in the build they installed. Feature work continues on a branch; the tagged artefact stays exactly reproducible.

Takeaways

1. If the catalogue is large, make the catalogue data. The question isn't "how do I build 500 tools" but "what abstraction makes the 500th cost one file".
2. Put execution strategy in the registry. Turning a store release into a config change is the highest-leverage indirection in the whole design.
3. A good abstraction refuses work. Widening the engine to fit a stopwatch would have cost every other tool its simplicity.
4. Budget 3–4 weeks for Play, and know that internal testing days don't count toward the 12-testers-for-14-days gate.
5. Your dependencies write part of your privacy policy. Read the merged manifest before you fill in Data Safety.
6. Record why you answered each declaration as you did. Reviewers ask, and the code will not remember.