Three ways to test the app you paid for

You have built an app, it works on your phone, and your developer says it has been tested. Before you put real customers and real money through it, you should know what kind of testing that was. Black box, grey box and white box testing look at your software from three different distances, and each one finds problems the others can miss.

The names describe how much the tester can see. A black box tester sees nothing inside. A white box tester sees everything, including the source code. Grey box sits in between.

Black box testing: the outsider's view

A black box tester uses your app the way a customer or an attacker would. They click buttons, fill in forms with odd data, and try to get into places they shouldn't. They never see the code.

The most familiar form is the penetration test, where a security specialist tries to break into a running system. The UK's National Cyber Security Centre calls this a "closed" or "opaque" box test when testers get no information about the system's internals.

Black box testing shows you what an outsider can do today. It has limits:

  • The testers only find what they can reach in the time you paid for. The NCSC warns that a lack of information "can also result in vulnerabilities remaining undiscovered in the time allocated for testing".
  • A test checks the system at one point in time. OWASP, the open-source security community, describes penetration testing as "generally a black-box point in time test".
  • The NCSC describes penetration testing as "not a magic bullet" and says it should not be your primary way of finding weaknesses.

Grey box testing: a guided tour

In a grey box test, you give the tester some inside knowledge: a login for each type of user, a diagram of how the system fits together, or notes on which features matter most.

That head start lets them skip guessing and spend their time on the risky parts. If your app has an admin area, a grey box tester can log in as an ordinary user and check whether they can reach admin pages they shouldn't see. For most founders launching a first product, this gives the best value from a security test of the live system.

White box testing: reading the code

A white box tester reads your source code. The NCSC calls this an "open" or "transparent" box approach, where testers get full information about the target.

Reading the code finds problems that never show up from the outside until the day they cause harm:

  • Security flaws hidden in paths a tester would never think to click, such as a forgotten test account or a password stored in plain text.
  • Performance problems, like a database query that runs fine with 50 customers and grinds to a halt with 5,000.
  • Bugs and fragile logic, such as an error that gets swallowed silently or a payment calculation that rounds the wrong way.

OWASP's Code Review Guide calls code review "probably the single-most effective technique for identifying security bugs early" in development. The same guide says that using it together with penetration testing can make your overall security testing much more cost-effective.

Which one does your app need?

Use the stage of your product to decide:

  1. Before launch, or after a big change: a white box code review catches problems while they are cheap to fix.
  2. Before handling sensitive data at scale: add a grey box penetration test of the live system.
  3. Once a year, or when a customer or insurer asks: a black box test confirms what an outsider can see.

If someone else built your app, or you inherited code from a freelancer or an earlier agency, a code review tells you what you have bought.

Synthetic Bytes offers a fixed-price code review and white box testing service, overseen by a senior engineer. You get a written report covering security, performance and bugs, with a prioritised plan for fixing what we find.

Sources