Research Gap, Work Undertaken and Contribution
Thesis: A Framework for Testing the Reliability of Generative AI for Penetration Testing Tasks
1. Research problem & gaps identified
Generative AI (GenAI) offers opportunities to automate penetration-testing activities that traditionally require substantial human knowledge and analytical skills. Studies showed that GenAI could perform various penetration-testing tasks but provided limited evidence of how consistently and correctly those tasks could be performed over repeated attempts.
In this research, reliability was defined as the extent to which GenAI consistently and repeatedly produces correct results under similar conditions when performing penetration-testing tasks.
Three principal gaps were identified:
Insufficient empirical investigation of reliability. Published studies provided evidence that GenAI could perform various penetration-testing tasks, but only a few examined the consistency and correctness of outputs over repeated attempts.
Lack of established testing approaches: There was insufficient guidance on how to design and conduct reliability tests for GenAI. Traditional software-testing approaches presented difficulties because GenAI outputs can vary, multiple answers may be acceptable, and determining the correctness of an output may require human judgement.
Lack of standardised benchmarking and reporting. Existing studies employed different testing approaches and reported results in different ways, making it difficult to compare findings and build cumulative evidence of reliability.
The gaps existed outside of security testing and findings may be generalised to other fields, with further research. The current research therefore investigated how the reliability of GenAI performing simple pen-testing tasks could be systematically evaluated.
2. Research undertaken
The research combined a literature review with iterative empirical experimentation to investigate the reliability of GPT-4o in performing selected penetration-testing activities. The literature review examined existing studies of GenAI for penetration testing alongside relevant research on software-testing challenges.
AI & Pentesting references [Lterature review]
Three suites of experiments were developed and conducted using GPT-4o and a controlled penetration-testing environment based on Metasploitable 2.
The experiments investigated the consistency and correctness of outputs when interpreting network scans, identifying services and vulnerabilities, and assessing security risks.
An iterative approach was adopted in which findings from earlier experiments informed subsequent test designs. This process revealed several challenges, including inconsistent outputs, omitted information, difficulties in evaluating correctness, and cases where the model appeared to reproduce or simulate repeated analyses rather than independently performing them.
The experiments also investigated the relationships between individual analytical tasks and sequences of tasks, informing the development of a proposed benchmarking approach.
3. Research contribution
The principal contribution of my research was a proposed framework for designing, conducting and reporting reliability tests of GenAI systems performing penetration-testing tasks. The applicability of this framework is wider than the current security context. The framework comprised the following three elements.
3.1 Guidelines for reliability-test design
A set of nine guidelines and considerations is proposed, informed by the literature review and empirical experimentation. These addressed issues including input and output design, prompt transparency, repeatability, model parameters, test evaluation, comparability, task decomposition and the risks of attributing human-like reasoning to GenAI systems.
The guidelines provide an initial structured approach to designing reliability tests while accounting for characteristics of GenAI that complicate conventional software testing.
3.2 Benchmarking approach and notation
A notation was proposed to describe tasks and capture reliability measurements in a consistent and comparable form. The approach introduced the concepts of micro-steps, aggregations and spans, distinguishing individual activities, sequences of activities and apparent direct transitions between stages of analysis.
A performance rating was also proposed to record successful outcomes relative to the number of attempts, providing a common basis for reporting and comparing experimental results.
3.3 A proposed grouping of penetration testing tasks
Penetration-testing activities were organised into four proposed blocks based on differences in task characteristics and experimental test findings.
This decomposition supports the design of manageable, repeatable experiments and enables individual activities to be evaluated without requiring repeated execution of an entire penetration test.
3.4 Empirical findings
The experiments generated reliability measurements for GPT-4o in selected penetration-testing tasks and identified inconsistencies relevant to test design.
The findings demonstrated the importance of independently repeated tests, careful evaluation of correctness and transparent reporting of experimental conditions.
They also provided practical evidence informing the development of the proposed framework.
4. Significance and limitations
The research made a methodological contribution by moving further towards a structured approach for evaluating the reliability of task performance. Its contribution lies primarily in the proposed testing framework, supported by empirical investigation and an initial application to penetration-testing tasks.
However, the framework was exploratory. The empirical work concentrated on one GenAI model and a restricted set of penetration-testing activities. The wider applicability and effectiveness of the proposed guidelines, benchmarking notation and task groupings have not yet been systematically established.
The research therefore provides a foundation for further investigation rather than a fully validated reliability-testing methodology.
5 Evolution of the research
The initial objective of this research was to assess the reliability of GPT-4o in performing penetration-testing tasks. However, the literature review and preliminary experiments revealed a more fundamental methodological problem: there was insufficient established guidance on how to design and conduct systematic reliability tests for generative AI.
In particular, the variable nature of GenAI outputs, difficulties in establishing correctness, and challenges in ensuring independent repetition raised questions about the suitability of conventional software-testing approaches.
Consequently, the research evolved from an empirical evaluation of a particular GenAI model into an investigation of how such evaluations should be conducted.
An iterative experimental approach was adopted, using GPT-4o as a test subject to investigate methodological challenges and develop a proposed reliability-testing framework. The principal contribution was therefore methodological: a set of test-design guidelines, a benchmarking notation and a proposed grouping of tasks to support systematic reliability evaluation. The GPT-4o performance measurements constituted supporting empirical findings rather than the primary research contribution
4. Significance, limitations & future directions
There is considerable scope to extend this work. The research made a methodological contribution by moving further towards a structured approach for evaluating the reliability of task performance.
The empirical work concentrated on one GenAI model and a restricted set of penetration-testing activities. The wider applicability and effectiveness of the proposed guidelines, benchmarking notation and task groupings have not yet been systematically established. The framework needs to be tested with larger numbers of tests and more complex penetration testing activities. Further investigation is also needed into the effect of model parameters and other variables on reliability.
Other research work suggesting the use of multiagent systems, agent-like components and AutoGPT may have the potential to go further in terms of automation. The area of multiagent AI is not yet developed enough to offer results yet but may benefit from reliability studies.
The research therefore provides a foundation for further investigation rather than a fully validated reliability-testing methodology.