Penetration testing is a useful tool to find vulnerabilities in software. But knowing its limitations is also important.

“Testing” in software systems means verifying the system works as expected. However, testing a system for security issues is a completely different beast. It is often a difficult challenge for developers, because it turns the testing concept on its head: instead of checking what the system should do, they need to verify that the system doesn’t misbehave when faced with unexpected situations and malicious input. But how is it possible to verify a negative? Penetration testing (also called pen testing and pentesting) tries to answer this question in a succinct manner via a short security evaluation where the testers take on the role of an external attacker.

Note that in this article we’ll only be talking about the software security aspect of penetration testing. A penetration test or “pen test” may have social engineering and physical security aspects, but they are testing the security of the company rather than the security of the product.

What You See Is What You Get

The concept of penetration testing comes from penetration exercises carried out by the US Department of Defense to uncover and fix vulnerabilities since the 1970s. From the get-go, penetration testing was a simulation of external attackers. A penetration test always implies that whenever a vulnerability is found, the testers create and execute an attack around it as well.

Pen testing, by design, is intended to be “lean and mean”, fitting into the DevOps paradigm: fast (and relatively cheap), utilizing automated tools as much as possible, and focusing on the externally facing components of the application. This, combined with simulating a real-world attack scenario, makes it appear to be effective – let’s look at its main benefits and downsides.

The Good, The Bad, and the Dangerous

Perhaps the best thing about penetration testing is that all its findings are explicitly proven by actual exploitation, and – depending on the skill of the testers – the test can approximate a real attacker. It is especially useful in network security and in identifying whether a particular infrastructure has components with known vulnerabilities. In an application security context, penetration testing is useful at finding the “low-hanging fruit”.

The weaker points of penetration testing are related to its relative inflexibility. The short timeframe and (usually) black-box approach are not conducive to finding ‘deep’ vulnerabilities in the business logic or in less-obvious interfaces that may not be directly exposed. Findings may also correlate with the power of the testing tools being used. Finally, the creation of working exploits as per penetration test requirements requires significant extra time. The average development time of an exploit is 22 days (see page 57 of ‘Zero Days, Thousands of Nights’). This may not be a problem for security researchers in general but can definitely be an issue in a one- or two-week penetration test – even for simpler exploits such as in web applications!

Over-reliance on penetration testing has some clear downsides. First, penetration tests – successful or unsuccessful – can instill a false sense of security, and an assumption that the test managed to identify most critical vulnerabilities. In addition, pen testing is a late lifecycle activity. By the time the testers execute the pen test, the program is basically finished. Therefore, if the test identifies any vulnerabilities in the code, fixing them will be significantly more expensive to fix than if the developers found the same vulnerabilities earlier during development. This goes double for design flaws such as storing passwords incorrectly.

Gary McGraw, a pioneer of the software security field and inventor of the Building Security In Maturity Model, has been particularly critical of these aspects of penetration testing in the past. The OWASP Security Testing Guide quotes him as such: “In practice, a penetration test can only identify a small representative sample of all possible security risks in a system.” In the quoted article, he concludes: “When a penetration test concentrates on finding and removing a small handful of bugs (and does so successfully), everyone looks good: the testers look smart for finding a problem, the builders look benevolent for acquiescing to the test, and the executives can check off the security box and get on with making money. Unfortunately, penetration testing done without any basis in security risk analysis leads to this situation with alarming frequency. By analogy, imagine declaring testing victory by finding and removing only the first one or two bugs encountered during system testing!”

Penetration testing and the real world

Pen tests can be useful for several things. They can serve as a baseline, they can identify low-hanging fruit, and act as an acid test to highlight glaring problems with the software. More importantly, they can accomplish this without needing too many resources.

Ideally, though, secure software development activities should be built into the SDLC, and penetration testing should be used as an additional layer of verification, not the be-all and end-all. We discuss building security into the software development process in some other articles in the context of software security and secure coding practices.

Of course, nobody knows your software better than your developers – it’s a good idea to have them act as your internal penetration testers by teaching them the mindset and skills required to do security testing. We offer several training courses to help you with this, whether your language of choice is Java, Python, C#, or C/C++.