CTF in secure coding education – a critical look

Competitive hacker events called CTFs (acronym for Capture the Flag) have been steadily gaining popularity in the corporate training world. They are meant to educate developers and cyber security experts about security in today’s age of digital transformation. But relying on CTF for this purpose does not provide an optimal return on investment and can even be contra-productive. CTFs focus on fun and flashy exploits over dealing with more mundane (but more important) best practices; thus, their usefulness in education is questionable. Also, they don’t work well for developers who aren’t already involved with software security. Most of all, CTFs are – by their nature – competitions, and not every programmer wants necessarily to ‘win’ against their peers; so, these events do not motivate everyone equally.

Secure coding is a team sport, and therefore education should reach out all developers! Let’s see what the difference is between doing CTFs and educating developers on software security.

What’s a CTF?

The term capture the flag originally referred to an outdoor sport where two opposing teams had to retrieve a flag from the other team’s “base” without being caught. In the 1990s, the hacker community adopted the CTF name for competitive hacking: an activity where multiple individuals (or teams) compete against each other in an event for points on a leaderboard. Such competitions are either done online, or in-person, typically in conjunction with cyber security conferences.

Generally, we can categorize CTFs depending on the type of activity they involve from the participants:

• Attack-only CTFs present several challenges to each participant (or team) where they have to find and exploit vulnerabilities to achieve various objectives such as reading the contents of a specific file on the target system’s file system. When they are successful, they obtain a ‘flag’ and gain points for it. These CTFs are often called ‘Jeopardy CTFs’ due to similarities with the popular TV quiz show “Jeopardy!”.
• Attack-and-defend CTFs involve two different teams who try to capture a target system by exploiting vulnerabilities in the software running on it, and then trying to harden it from attacks by the other team at real time (without applying obvious countermeasures such as removing the vulnerable components completely).
• Defend-only CTFs have the participant try to secure a system from attack by applying hardening and best practices. Some of these can be considered a subtype of attack-and-defend CTFs where the organizers make up the ‘attack team’, or the attack scenarios are entirely simulated. These are more relevant in a corporate context as opposed to public hacking competitions – in general, players perceive pure defense to be ‘less fun’ and ‘more annoying’.

CTFs and CTF-like training platforms – also called ‘cyber ranges’ in the corporate training world – have been gaining popularity over the last decade, primarily for educating cyber security experts.

But how applicable are CTFs when it comes to training developers in secure coding?

What CTFs are great at

Let’s take a look at the strengths of CTFs first by going through the arguments presented from some publications on the subject, both from the early 2010s – such as Hacking Competitions and Their Untapped Potential for Security Education (2011), Winning Cybersecurity One Challenge at a Time (2012) – and newer ones such as Gamifying ICS Security Training and Research: Design, Implementation, and Results of S3 (2017) and Supporting Cybersecurity Education and Training via LMS Integration: CyLMS (2019). Some of these articles glorify the usefulness of capture the flag in academia, and yet others emphasize their usefulness for cyber security professionals.

First of all, CTFs are fun thanks to gamification. They provide novel, interesting hacking challenges to participants, and the feeling of competition may be a strong motivator. This can help a lot with engagement, especially in the academic context.

Furthermore, instead of just providing theoretical knowledge, CTFs present realistic scenarios that participants have to overcome – this both helps with driving home the real dangers posed by cyber criminals, as well as honing practical skills in an environment that is much closer to real systems than the simplified examples shown in traditional training courses.

Finally, arguably the most important thing in cyber security is to get an insight into the “hacker mindset”: an intuition to see potential weak points in a system and how they could be exploited. This is very useful not just for hackers – arguably it’s just as important for developers so they can see potential security holes in their (and of course in their colleagues’) code and fix them yet at development time.

For this reason, public capture the flag competitions are very popular among cyber security experts – in addition to recognition and fame, good results from CTFs can be useful when hunting for jobs or scholarships too, especially when a participant writes a detailed solution (called a ‘writeup’) of a particular task after the CTF to share their knowledge with others. The ctftime.org website keeps track of all team standings, enlisting around 30 thousand teams worldwide. Even considering that many players are in multiple teams, that is a large community; but it is still negligible compared to some 25 million developers we have worldwide.

All in all, CTF popularity is surging, and more and more cyber security training platforms implement them. But what are the downsides?

What CTFs are less great at

By far, the biggest limitation of the CTF approach is how well the skills learnt during a challenge translate in everyday work. For cyber security experts and ‘security champions’, the benefit is obvious. By overcoming difficult hacking challenges, they can hone their skills in a lot of fields related to their work: finding and exploiting vulnerabilities, reverse engineering code, and evading various protection techniques. However, for developers, the connection is less clear.

The efficacy of CTF in learning has been a topic of numerous studies in industry and academia. Here is a list of quite a few downsides they’ve discovered.

• Cybersecurity Knowledge and Skills Taught in Capture the Flag Challenges analyzed existing CTF challenges for coverage of cybersecurity topics and found that some topics (such as cryptography) are overrepresented while others (such as privacy issues and GDPR compliance) are underrepresented compared to their real-world relevance. This is not surprising when considering that some of these tasks are more fun and challenging to do than others. For a practical example: at the time of writing, when combining all subtypes of integer overflow vulnerabilities (which are reasonably common in C and C++ in practice), ctftime.org showed only 17 tasks in total. On the other hand, there are over 100 matches when searching for tasks about the use of externally-controlled format string vulnerability (which is more fun, but is an extremely rare issue these days, especially in modern C and C++ code; even if it recently popped up in automotive security). We admit that exploiting an integer overflow may be mundane and ‘boring’ while exploiting a format string vulnerability is exciting and challenging. But the statistics clearly show that knowledge of integer overflows is still vastly more important!
• Benefits and Pitfalls of Using Capture the Flag Games in University Courses showed the positive effects on engagement and fun, but also identified weaknesses in the hint systems of CTF platforms, such as the inability to adapt hints to the different needs of each participant. The study also showed that the scoreboard did not particularly motivate the participating students. This highlights another important point that goes beyond university education. Competitiveness obviously does not motivate everyone equally, especially if participation is mandatory as part of the company training curriculum.
• As per CTF: State-of-the-Art and Building the Next Generation, “When competitors do not have sufficient background to compete, as is the case when introducing individuals to completely new concepts and topics, competition can have a negative effect on educational outcomes.” Thus, the competitive and gamified nature of CTF can in fact have a negative effect for programmers who are less skilled or yet unfamiliar with cyber security concepts and issues.
• In the context of attack-defense CTFs, The Fun and Future of CTF does mention that “The time frame restriction may make it difficult to employ slow stealthy attacks or social engineering techniques […], and may encourage manual defensive solutions that do not scale well to real-world problems”. Though the conclusion is “[…] that despite these artifacts, attack-defend CTFs represent a reasonably realistic laboratory”, all this highlights that the event-oriented nature of CTF makes the tasks unrealistic in several ways due to the compressed time frame.

Can it work for secure coding?

Given the limitations of CTF, its applicability ‘as-is’ is rather narrow when it comes to secure coding. It is basically only suitable for security champions. But that doesn’t mean the same approach can’t be used to make secure coding education more engaging.

When adapting capture the flag for secure coding, a critical point is de-emphasizing competition and focusing on providing help and guidance to participants who need it. A platform that was built for education in the first place will have the necessary facilities to do this already.

Knowing about attacks is important only so that developers become aware of the consequences of insecure code. But they certainly do NOT need to know about in-depth exploitation. An injection vulnerability is an injection vulnerability; whether it can be exploited via a simple payload or if it needs a complicated multi-stage exploit and possibly some evasion of firewall rules is something that an average developer does not need to worry about! What is important to them however is to understand how to avoid and fix the problem, what the available protections are, and what are the weaknesses or tradeoffs involved with these protections. At some level, the conceptual difference is similar to that between penetration testing and in-depth security evaluation.

The usefulness of capture the flag in secure coding education – at this point – has not been proven one way or the other. There has been some movement in the CTF arena in this direction recently, however, with some important research identifying the requirements for a CTF system to be able to properly support industry requirements (e.g. On the Requirements for Serious Games geared towards Software Developers in the Industry).

Points where CTFs tend to be weak include, among others:

• Questionable knowledge retention due to lack of didactics;
• Insufficient adaptation to different skill levels and backgrounds of developers;
• Poorly-defined working mechanics that often rely on specific knowledge of hacking tools and (sometimes) obscure knowledge;
• Lack of adaptation to company internal secure coding guidelines; and
• Lack or scarcity of defensive challenges.

One important aspect is highlighting the importance of the (usually neglected) defensive CTF. An example of adapting capture the flag for this purpose is Build It Break It Fix It (BIBIFI for short). In this contest, developer teams first have to build software according to a specification, then try to find weaknesses in others’ submissions, and then finally fix all issues found in their software by the other teams.

Another aspect of CTFs that have to be adapted is the mindset of viewing it as a single competitive event that is meant to ‘kickstart’ security knowledge in a company. Doing a CTF is fun, but it is still essentially a single team-based competition event as opposed to a long-term education plan. And while exploiting a particularly tricky format string vulnerability may give a developer a great sense of accomplishment, knowing how to systematically apply input validation and defensive programming techniques is far more valuable in the long run.

Code responsibly! Learn approaches, get skills, and do drills! With the help of our unique Cydrill Sergeant learning environment, everyone will get the right guidance in their own context and at the right time. Get in touch with us and check out how.

See the courses in our catalog. We cover all popular programming languages and platforms . Pick the most appropriate course for your development group and let us know how you want us to deliver it: instructor-led online or on-site, or as e-learning.