1.0    Introduction

Almost 20 years after the first publicized buffer overflow vulnerability and exploit used by Morris worm in November, 1988 [1] then followed by Code Red worm in July 2001 [2] and Slammer worm [3] in January 2003, the buffer overflow still one of the top vulnerability, proven to have a severe effect, which has been exploited successfully. Since then, the buffer overflow vulnerable exploited again and again which cover a wide range of computer application, library, operating system (kernel and embedded system as well) and networking.

There are many hardware and software based techniques and tools that have been proposed and developed to detect and/or protect from buffer overflow vulnerable. However this vulnerable still happens and based on the trend it look likes this problem will continue to happen.

It is publicly known that the buffer overflow happens when there is no bound checking on the used buffer in programs. In the plain source codes, this no bound checking is a normal for unsafe programming language that dominated by C and C++ for a set of the standard library functions. Unfortunately, C and C++ are the languages that most widely used for critical applications such as kernel, Operating System (OS), database engine and device driver.

A buffer is small and reusable temporary data storage during the program execution. Normally it is declared as sized array data type in C and C++ programs though it is not limited to the declared sized array because other unsafe standard C and C++ functions used for string and character manipulation such as strcat() and gets() also resembled similar characteristics. Without bound checking, input size that bigger than the declared size of the array will overwrite other adjacent data in memory and corrupting them. From the programmer point of view, when declaring a sized array in a program, we normally already expected or pre-calculated the maximum number of input. However, we cannot accurately determine the maximum number of input from user and other applications which will use the program. This input validation is more prevalent for applications that will call and share the program such as DLL.

With freely available tools and proof-of-concept (POC) codes, on the attacker side, without proper user input validation, it looks like it is easy to exploit the buffer overflow vulnerability. Furthermore, user input validation issue already exploited by other code injection exploit methods such as SQL injection.

In finding the root cause, it is quite obvious that buffer overflow vulnerable is not a simple thing for the average person to understand. It is not just a programming flaw that represents programmer’s competency to solve it. Much more knowledge such as processor architecture, memory layout and operation, library, how compiler works and low level programming in certain circumstances, must be fully understood in order to detect and protect from buffer overflow effectively.

1.1    Scope and Limitation

The scope of this for fun study is limited to the stack-based buffer overflow with local exploit. All the hardware and related software used in the controlled demonstration can be found in Chapter 3. Keep in mind that there are several buffer overflow variants such as heap based, integer, off-by-one and also a similar subset found in format string attack.