Skip to content

Learning in Public: Why We Publish How We Test, Not Just That We Do

23 September 20267 min readBy , LeelaRam Technologies
Share

'Accurate' and 'reliable' are claims anyone can make. Publishing the method turns a claim into something a customer can check — and turns marketing into a forcing function for better engineering.

Every software product page says accurate, reliable and secure. The words are free, so they carry no information. The only version of those claims worth anything is one where the method is published and the customer could, in principle, run it themselves.

The difference between a claim and a method

'Our sorting is accurate' is a claim. 'Every ranking option is verified against known inputs; the move planner is tested across more than 78,000 cases; after each live sync the app reads the positions back from Shopify and confirms they match what was computed' is a method. The second one can be wrong, which is precisely what makes it worth saying.

A claim that cannot be falsified is not reassuring. It is just noise that resembles reassurance.

Publishing changes what you build

This is the part that surprised me. Committing to describe how something works publicly changes how you build it, because you discover the parts you would be embarrassed to write down. A shortcut is easy to live with privately and difficult to put in a sentence.

Writing the piece about how we test the sorting engine meant reviewing what the suite actually covered. Some of it was thinner than the story I had been telling myself. The article got better because the suite got better first — which is the right order, and only happened because of the obligation to publish.

It is also the honest form of marketing

A small company cannot win on brand recognition or sales volume. What it can do is be checkable. Publishing methods, limitations and the reasoning behind design decisions lets a prospective customer evaluate the work rather than the pitch. The people persuaded by that tend to be the people worth working with.

It filters in both directions. Someone who wants confident claims and no caveats will find us unsatisfying, and that is a good outcome for everyone involved.

Publishing limits is more persuasive than publishing strengths

Every honest product has edges. Ours reports zero margin where a cost price is not recorded, rather than estimating one, and the option is labelled estimated on the product page for exactly that reason. Products added to a collection between syncs sit wherever Shopify puts them until the next sync. A collection under SellRank is on manual sort, which is how the order is written at all.

Stating those plainly costs a few sales to people who wanted a different product, and saves a great deal of support and disappointment later. It also makes the positive claims credible, because a document that admits limits reads differently from one that does not.

What this looks like as a practice

  • Write the method, not the adjective. If you cannot describe how it is verified, do not claim it is verified.
  • Publish the limits alongside the capabilities, in the same place, at the same size.
  • Keep a changelog with dates, so claims about improvement are checkable.
  • When something goes wrong, describe the cause and the fix rather than the apology.
  • Prefer numbers a reader could reproduce over numbers that merely sound impressive.

The underlying bet

The bet is that over a few years, being checkable beats being loud. Customers who chose you because they could verify the work stay longer, ask better questions and make the product better. That is a slower way to grow, and it is the only one that compounds.

Sumit Chavda

Written by

Founder, LeelaRam Technologies

Founded LeelaRam Technologies after building and running e-commerce technology for a live jewellery brand. Builds SellRank, and writes here about Shopify, merchandising and software that keeps working.

LinkedIn Portfolio All articlesAbout the company

Common questions

Why publish how a product is tested?+

Because 'accurate' is a claim anyone can make and a method is something a customer can check. Publishing the method also improves the engineering, since committing to describe it publicly exposes the shortcuts you would not want to write down.

Does admitting product limitations hurt sales?+

It costs some sales to people who wanted a different product, and saves considerable support and disappointment later. It also makes the positive claims credible, because a document that admits limits reads differently from one that does not.

How can I verify SellRank's accuracy claims?+

The method is published: every ranking option is verified against known inputs, the move planner is tested across more than 78,000 cases, and after each live sync the app reads positions back from Shopify to confirm they match what was computed.

Related

Put this to work on your store

SellRank orders any Shopify collection by real demand — free to try on one collection, no code.

See the product