DevSecOps Metrics

DevSecOps is an emerging paradigm that breaks the Security Team Silo into the DevOps Methodology and adds security practices to the Software Development Cycle (SDL). Security practices in SDL are important to avoid data breaches, guarantee compliance with the law and is an obligation to protect customers data. This study aims to identify metrics teams can use to measure the effectiveness of DevSecOps methodology implementation inside organizations. To that end, we performed a Multivocal Literature Review (MLR), where we reviewed a selection of grey literature. Several metrics purposed by professionals to monitor DevSecOps were identified and listed.


Introduction
Nowadays there is a trending methodology within Information Technology (IT) called DevOps that from a highlevel perspective is defined has the merging of the Development team and Operations team into one. This methodology has proven productivity gains and DevOps professionals feel their work has more impact and it's recognized by all the organization [1]. DevOps increases both deployment frequency and the pace by which companies can serve their customers without compromising the quality of deliveries [2] . DevOps has indeed influenced software development but faster development cycles and increase of deployments that DevOps promises in conjunction with new engineering practices and tools may compromise security and this is discussed on research related with security aspects of DevOps [3] other research focus on security on CI/CD pipeline [4] from these researches the term DevSecOps and other aliases were coined [2]. DevSecOps is defined as the integration of security practices into DevOps [5]. This term is still recent but already is consider has topic having its own merit [2].
This research aims to study the scientific developments on DevSecOps and elicit a set of metrics grounded on professional and academics viewpoints, so organizations can monitor DevSecOps. Metrics are important to improve the rigor of measurement in both Software Engineering and Information systems fields and proposing such measures opens a debate for better understanding of the topic under discussion [6].
Since DevSecOps is a very recent topic the research methodology selected for this study is a MLR. MLR is a kind of Systematic Literature Review (SLR) [7] and is useful when trying to close the gap between academic research and professional practice [8].
The rest of this document is organized as such. Chapter 2 gives theoretical background on DevOps and DevSecOps, Chapter 3 describes the research methodology, Chapter 4 describes the literature review plan, Chapter 5 summarizes the information extracted from the analyzed publications, and discusses the results and limitations of the study, and Chapter 6 reports the findings and Chapter 7 concludes the paper.

DevOps
DevOps literature shows that defining the term has been hard. DevOps most typical description is Development plus Operations, but this description is not enough to explain DevOps [9]. Roche provides a good summary on the different viewpoints of what is DevOps. For some it is a specific job that requires development and IT operational skills for others DevOps is more than that [10]. Those who think that the term is more than a specific job defend the existence of four perspectives: collaboration, automation, sharing and measurement [11] [12].
DevOps is not only culture aspects it is also a set of engineering practices influenced by cultural aspects and supported by technological enablers [9]. DevOps capabilities are Continuous planning, Continuous integration and testing, Continuous release and deployment, Continuous infrastructure monitoring and optimization, Collaborative and continuous development, Continuous user behavior monitoring and feedback [9] [13] .
DevOps is a complete new organizational mindset that replaces siloed units with cross-functional teams. DevOps achieves this by taking advantage of automated development, deployment, and infrastructure and enables teams to continuous work and deliver operational features [14].

DevSecOps
The same way that we can say DevOps is Development and Operations merged together we can say that DevSecOps is Development, Security and Operations merged together. DevSecOps is defined in literature as the integration of security processes and practices into DevOps environments and seen as a necessary expansion to DevOps [5].
The terms "DevSecOps", "SecDevOps", "SecOps", "RuggedOps", "Security in Continuous Delivery", and "Security in Continuous Deployment" are all aliases to DevSecOps [3]. In current literature is already possible to find a set of practices for DevSecOps [5]. Continuous Testing, Security as Code, Threat modelling, Risk analysis, Monitoring and logging and Red Team security drills. Continuous Testing is the practice of having automatic security controls throughout the software development lifecycle, continuously detecting for defects in code changes with the possibility of automatic rollback if necessary [13] [5]. Security as Code is the practice of having security policies like network configurations codified integrated with software development lifecycle [5].
Monitoring and logging practices is observing various quality parameters associated with the implemented controls and measure their effectiveness [5] [13]. Threat Modeling is the activity attacking your system on paper and using this information to identify, describe, and categorize threats to your system [3] [5]. Risk Analysis is the activity of creating security design specifications from the first planning and before every iteration [3] [5]. Red Team security drills is the practice of creating a proactive team that performs a malicious attack on deployed software with the intent of finding and exploiting vulnerabilities, finding security flaws and helping the organization find solutions [5] [15].
The two main benefits of DevSecOps are having fast and scalable security controls by Automating Security and having security controls since the beginning of the development process by Shifting Security to Left, this means bringing security experts involved from the beginning to plan and integrate security controls [5] but also to share knowledge with other team elements making them more security aware.

Research Methodology
This study follows a MLR methodology. A MLR is a form of a SLR which includes grey literature in addition to the published (formal) literature [7].
MLR in Software Engineering (SE) is not usual [7] and there are no guidelines to perform a MLR, since MLR is a form of SLR the review is planned as SLR but including "grey literature".
SLR is a type of literature review that is used to identify, evaluate and interpreting all available research relevant to a specific question [16]. Kitchenham's procedures for performing systematic reviews will be adopted by the authors. Error! Reference source not found. Fig. 1 details how this research steps maps to the three phases proposed by Kitchenham [16].

Fig. 1. MLR Steps
Planning Reviewthis phase consists in three steps. First step is identifying the need and motivation for the review, second step is specifying the research questions that are going to be addressed and answered by the review.
Final step designing a review protocol with the constraints that are going to be applied in the review. This phase is presented in Section 4.
Conducting Reviewthis phase consists in applying the designed review protocol. This phase is presented in Section 5.
Reporting Reviewfinal phase of the review is summarizing the extracted data from the selected literature and report findings. This phase is presented in Section 6.

Planning the Review
This section details the first phase of the SLR. Motivation for this work is presented, followed by the Research Question this study intent to address and answer. Finally, Review Protocol is proposed.

Motivation
This research aims to study the scientific developments on DevSecOps and elicit a set of metrics grounded on professional and academics viewpoints, so organizations can monitor DevSecOps. Metrics are important to improve the rigor of measurement in both Software Engineering and Information systems fields and proposing such measures opens a debate for better understanding of the topic under discussion [6]. One of the principles found in DevOps and DevSecOps is measuring. DevSecOps encourages development of metrics that track threats and vulnerabilities throughout the software development lifecycle. Applying automatic security controls to the software development process provides development teams with metrics capable of tracking threats and vulnerabilities, allowing the organization with insights on the quality of software being developed [5].
Therefore, this work aims to obtain information about which metrics associated with DevSecOps are already identified by academics and professionals and the value they bring to development teams and organizations.

Research Questions
Based on what was described before it was established the importance of having metrics has way to better understand a topic under discussion for that reason the research aims to answer the following Research Question (RQ).
RQ: Which are the most relevant DevSecOps metrics.

Review Protocol
The first stage of the review protocol is literature search, a search string must be defined and applied in the chosen data sources with the intent of retrieving the highest possible number of studies related with the proposed research questions.
The search string is a set of keywords related to DevSecOps. Search terms used in this research are presented in Table 1. Inclusion and exclusion criteria is applied to literature from both data sources. Criteria is presented in Table 2.

-Conducting the Review
This section corresponds to second phase of the MLR and consists of applying the previously defined review protocol.

Selection of Studies
First step was to run the search string composed by the search terms defined on Table 1. After running the search terms on the selected data sources 558 articles were obtained. Distribution of articles by category is illustrated on  Next step of the review protocol is applying the inclusion and exclusion criteria.

Academic Databases.
First step is ensuring that there is not duplicated articles. Removing the duplicates consists on a two-step approach.
1. Remove Duplicates from articles retrieve from same database.

Remove Duplicates between the four academic databases.
Studies information exported from each data source were on different formats. Table 3 shows the export format from each academic data source.    all texts were read to further decide the document's relevance, and a total of 13 were obtained as relevant to our study.

Data Extraction Analysis
Based on the obtained artefacts from this MLR there is little literature related with DevSecOps and in particularly on literature related in how organizations can measure the efficiency of DevSecOps implementations.
Even so we can verify that the topic has been gaining interest as it can be seen in Fig. 5, both in academic and grey literature data sources the interest on the topic rose considerably after 2017.  Final selection of studies only contained 2 academic articles and much of the literature use to answer the research questions is based on Blogs and articles from industry professionals. Table 6 summarizes number of articles based on literature source.

Reporting the Review
This MLR phase presents the research done on DevSecOps to identify its metrics. We used Google Scholar, Google Search, IEEE Explore, Springer and ACM Library to locate literature and after applying our inclusion and exclusion criteria, 15 articles were found to be relevant to our search terms. Only 2 of those were academic research papers. The remaining 11 consisted of blogs and articles. Based on the literature review found e 9 relevant metrics were reported by professionals. Table 7 lists and describes identified metrics.

Conclusion and Future Work
This MLR presents the research done on DevSecOps to identify metrics associated with DevSecOps that can be used to measure its effectiveness. DevSecOps is a recent topic has it was established earlier it is expected to continue to grow. It was very hard to find information regarding metrics associated with DevSecOps special in academic literature. Even so it was possible to identify a total 9 metrics as indicators of DevSecOps effectiveness.
This topic is expected to grow and for that reason it's study should continue; this study serves as the initial support for further studies.
This study for academics may serve has the basis for further research into DevSecOps metrics or other related metrics. For Professionals this study summarizes the principal metrics for measuring DevSecOps effectiveness in one document.
Since DevSecOps is a trending topic and this study had an exploratory nature, further researches may continue the study performing interviews and surveys with DevSecOps professionals to tune and complement the proposed metrics as well as what is the outcome of each one. Plus, it would also be interesting to understand what mechanisms and policies could be implemented to mitigate the security issues that the presented metrics are intended to measure. The authors are already pursuing this investigation line. Identifies how many adversaries an application might have this metric is associated with the practice of Threat Modelling and Risk Analysis. The goal is to identify the applications inside an organization that are more exposed to possible attacks and prepare accordingly. Team exercise where the objective is to think how many adversaries they think an application as and register those findings.

Adversary Return
Rate [17] Measures how often an adversary will use the same strategy and procedures.
Helps define appropriate training to better handle these attacks. Measure is done by counting the number of times adversaries use the same attacking strategy and compiling into a ranking that visible for every team member. Ideal is to have a plan to handle each attacking strategy.
Point of Risk Per Device [18] Tracks the number of vulnerabilities per server.
Helps prioritize these vulnerabilities according to their criticality giving special attention to the ones that are most exposed to attack from the internet. Identify and keep track of unpatched vulnerabilities per server. The number of vulnerabilities should tend to zero.