Structured Problem Solving Techniques

• Life Data Analysis (Weibull Analysis)

An Overview of Basic Concepts
In life data analysis (also called "Weibull analysis"), the practitioner attempts to make predictions about the life of all products in the population by fitting a statistical distribution to life data from a representative sample of units. The parameterized distribution for the data set can then be used to estimate important life characteristics of the product such as reliability or probability of failure at a specific time, the mean life and the failure rate. Life data analysis requires the practitioner to:

Gather life data for the product.
Select a lifetime distribution that will fit the data and model the life of the product.
Estimate the parameters that will fit the distribution to the data.
Generate plots and results that estimate the life characteristics of the product, such as the reliability or mean life.

https://www.weibull.com/basics/lifedata.htm

• Fault tree analysis (FTA) is a top-down, deductive failure analysis in which an undesired state of a system is analyzed using Boolean logic to combine a series of lower-level events. This analysis method is mainly used in the fields of safety engineering and reliability engineering to understand how systems can fail, to identify the best ways to reduce risk or to determine (or get a feeling for) event rates of a safety accident or a particular system level (functional) failure

https://en.wikipedia.org/wiki/Fault_tree_analysis


The fault tree analysis (FTA) was first introduced by Bell Laboratories and is one of the most widely used methods in system reliability, maintainability and safety analysis. It is a deductive procedure used to determine the various combinations of hardware and software failures and human errors that could cause undesired events (referred to as top events) at the system level.

Fault tree construction
To do a comprehensive FTA, follow these steps:

Define the fault condition, and write down the top level failure.
Using technical information and professional judgments, determine the possible reasons for the failure to occur. Remember, these are level two elements because they fall just below the top level failure in the tree.
Continue to break down each element with additional gates to lower levels. Consider the relationships between the elements to help you decide whether to use an "and" or an "or" logic gate.
Finalize and review the complete diagram. The chain can only be terminated in a basic fault: human, hardware or software.
If possible, evaluate the probability of occurrence for each of the lowest level elements and calculate the statistical probabilities from the bottom up.
http://asq.org/quality-progress/2002/03/problem-solving/what-is-a-fault-tree-analysis.html

• when a problem occurs, you drill down to its root cause by asking "why?" five times.

Then, when a counter-measure becomes apparent, you follow it through to prevent the issue from recurring.

The 5 Whys uses "counter-measures," rather than solutions. A counter-measure is an action or set of actions that seeks to prevent the problem arising again, while a solution may just seek to deal with the symptom. As such, counter-measures are more robust, and will more likely prevent the problem from recurring.

You can use 5 Whys for troubleshooting, quality improvement and problem solving, but it is most effective when used to resolve simple or moderately difficult problems.

5 Whys is an iterative interrogative technique used to explore the cause-and-effect relationships underlying a particular problem

The vehicle will not start. (the problem)

Why? - The battery is dead. (First why)

Why? - The alternator is not functioning. (Second why)

Why? - The alternator belt has broken. (Third why)

Why? - The alternator belt was well beyond its useful service life and not replaced. (Fourth why)

Why? - The vehicle was not maintained according to the recommended service schedule. (Fifth why, a root cause)

Two primary techniques are used to perform a 5 Whys analysis:

the fishbone (or Ishikawa) diagram

a tabular format

These tools allow for analysis to be branched in order to provide multiple root causes.

Criticism

Tendency for investigators to stop at symptoms rather than going on to lower-level root causes.

Inability to go beyond the investigator's current knowledge - cannot find causes that they do not already know.

Lack of support to help the investigator ask the right "why" questions.

Results are not repeatable - different people using 5 Whys come up with different causes for the same problem.

Tendency to isolate a single root cause, whereas each question could elicit many different root causes.

https://en.wikipedia.org/wiki/5_Whys

https://www.mindtools.com/pages/article/newTMC_5W.htm

• When Should I Use Structured Problem Solving?

It is important to use the right tool for the task at hand. This is a powerful method that takes some time to plan and use. As a result, it only makes sense to use it on medium or large problems.Finally, this process is useful to apply when you are facing an unfamiliar or frustrating problem.

The 6 Step Process For Problem Solving

Step 1: Identify the problem

2. Structure the problem

3. Develop solutions

4. Select a solution to the problem

5. Implement a solution

6. Monitor for success

http://projectmanagementhacks.com/how-to-use-structured-problem-solving/

• five problem-solving tools that can each be used to look at a particular problem from a different perspective

1-Six-Step Problem Solving Model 

Each step must be completed before moving on to the next step. However, the steps are repeatable. At any point the group can return to an earlier step, and proceed from there

The process is one of continuous improvement. The goal is not to solve but

to evolve, adjusting the solution continually as new challenges emerge,

through repeating the Six Step Process. 

Step One: Define the Problem

At this stage groups will use techniques such as:  Brainstorming  Interviewing  Questionnaires

Step Two: Determine the Root Cause(s) of the Problem

In this step the problem solving team will use tools such as:  Fishbone diagrams  Pareto analysis  Affinity diagrams

Step Three: Develop Alternative Solutions 

Techniques include:  Force field analysis  SWOT  Porters five forces 

Step Five: Implement the Solution 

The group may use tools, such as a Gantt chart, timeline or log frame 

2-The Drill Down Technique 

 Drill down is a simple technique for breaking complex problems down into progressively smaller parts. 

- Start by writing the problem down on the left-hand side of a large sheet of paper. 

- On the right of each point, write down the points that make up the next level of detail. 

- Repeat this process, for each new point that you identify. 

- Keep on drilling down until you have identified all of the factors contributing to the original problem. 

- This technique can be used in conjunction with the 5 Why Analysis to ensure that you investigate each aspect of the problem.

You can choose to work through this process either on your computer or with a pen on a piece of paper;

To start, write down the problem that you are facing in big letters at the top of the page. Try to sum up the problem in just a word or a short phrase, even if it is complicated in nature.

you are going to break down the problem into three to five smaller issues that make up the big problem.

This process will continue until you simply can’t drill down any farther. Once you have reached what you consider to be the bottom of your chart, you will be finished and you can begin to look for solutions among what you have created.

Specifically, the five whys method matches up with this line of thinking in a number of ways. Both methods are focused on getting to the heart of the problem rather than just fixing the top level issue, and both methods ask you to think about the operation of your business as a whole, instead of just the factors immediately related to the problem in front of you.

3-The Four Frame Model

The Four Frame Model is designed to help you understand and approach issues about organizational problems, development, and change.

It views organizations in four frames representing separate metaphors: structural (factories or machines), human resource (personal relationships), political (jungles or battles for power), and symbolic (theatre or drama). 

Each of these frames can be thought of as a different perspective or way of looking at things, which can help you to see the same situation in a variety of ways. 

The structural frame focuses on the architecture of the organization. This includes goals, structure, technology, roles and relationships. 

The human resource frame emphasizes individual needs, feelings, fears, prejudices, skills, and development opportunities. 

The political frame emphasizes power and competition, taking into account diverse beliefs, interests, behaviors, and skills.

The symbolic frame treats organizations as theatre or drama focusing on meaning and faith. 

With each of the four frames, the interested organizational observer can view the same situation in at least four ways. 

4-The Eight Disciplines Problem Solving

The Eight Disciplines Problem Solving procedure is focused on product and process improvement, its purpose is to identify, correct, and eliminate recurring problems.

It aims to establish a permanent corrective action based on fixing the origin of the problem by determining the root cause. 

Once a problem has been recognized, the 8 disciplines used to solve; 

Team Formation, Problem Description, Implementing Interim Containment Actions, Defining Problem Root Causes, Developing Permanent Corrective Actions, Implementing Permanent Corrective Actions, Preventing Reoccurrences, and Recognizing and Congratulating the Team.  

Once the problem has been resolved, the team should publish and release a final report along with lessons learned. 

5-The Cynefin Framework

The Cynefin Framework helps you figure out how you should be thinking about a problem rather than providing a method for solving it.

The core of this framework is the way that it breaks down problems into one of five contexts. 

The idea is to place the problem that you are facing into one of these specific contexts, which will then help you decide how that problem needs to be approached. 

The five contexts are: Obvious, Complicated, Complex, Chaotic and Disorder. 

Obvious: Are self-explanatory and the cause and effect relationships that you need to uncover are right there for you to see.

Complicated: Are those that are usually best left to experts in the specific field in question. 

Complex: Might not have a clear solution at the present time. You don’t necessarily need an expert in order to solve this problem; you may just need more time and information. 

Chaotic: There is no obvious connection between cause and effect. Once action has been taken and the problem has been mitigated as thoroughly as possible, you can then work toward removing the chaos and gaining a better understanding of what is going on. 

Disorder: The state of not knowing what type of causality exists, in which state people will revert to their own comfort zone in making a decision. 

Once you identify where in this framework your problems are found, you can then start to solve them using a variety of other means and methods. 

http://www.free-management-ebooks.com/dldebk-pdf/fme-top-5-problem-solving-tools.pdf

• a conceptual framework used to aid decision-making

it has been described as a "sense-making device"

Cynefin is a Welsh word for habitat

Cynefin offers five decision-making contexts or "domains"—obvious (known until 2014 as simple)

complicated, complex, chaotic, and disorder—that help managers to identify how they perceive situations and make sense of their own and other people's behaviour.

https://en.wikipedia.org/wiki/Cynefin_framework

• Non-abstract Large System Design Interview Preparation (My Path to SRM)

NALSD describes a skill critical to SRE: the ability to assess, design, and evaluate large systems. Practically, NALSD combines elements of capacity planning, component isolation, and graceful system degradation that are crucial to highly available production systems.

Google SREs are expected to be able to start resource planning with a basic whiteboard diagram of a system, think through the various scaling and failure domains, and focus their design into a concrete proposal for resources

    Design an image sharing service like Imgur and come up with a bill of materials for serving 50.000 Queries per second (QPS).

    Design a log ingestion service like Stackdriver including indexing pipeline and frontend. Highlight the tradeoff differences of the components.

    Design an approach to distributed rate limiting of an API that can handle one million QPS hitting the API endpoints. Optimize for less cross-regional bandwidth and come up with a reasonable bill of materials.

https://danrl.com/blog/2019/path-to-srm-nalsd/

• Introducing Non-Abstract Large System Design

By following an iterative style of system design and implementation, we arrive at robust and scalable designs with low operational costs. We call this style Non-Abstract Large System Design (NALSD). 

NALSD describes a skill critical to SRE: the ability to assess, design, and evaluate large systems. Practically, NALSD combines elements of capacity planning, component isolation, and graceful system degradation that are crucial to highly available production systems. Google SREs are expected to be able to start resource planning with a basic whiteboard diagram of a system, think through the various scaling and failure domains, and focus their design into a concrete proposal for resources. Because these systems change over time, it’s vitally important that an SRE is able to analyze and evaluate the key aspects of the system design.

Why “Non-Abstract”?

All systems will eventually have to run on real computers in real datacenters using real networks. 

AdWords Example

The Google AdWords service displays text advertisements on Google Web Search. The click-through rate (CTR) metric tells advertisers how well their ads are performing. CTR is the ratio of times the ad is clicked versus the number of times the ad is shown.

Design Process

Google uses an iterative approach to design systems that meet our goals

the NALSD process has two phases, each with two to three questions.

In the basic design phase, we try to invent a design that works in principle. We ask two questions:

Is it possible?

Can we do better?

In the next phase, we try to scale up our basic design—for example, by dramatically increasing a requirement. We ask three questions:

Is it feasible?

Is it resilient?

Can we do better?

With these concepts in mind, let’s walk through the iterative NALSD process.

Initial Requirements

when iterating on the design, we will consider our requirements in terms of SLOs

https://landing.google.com/sre/workbook/chapters/non-abstract-design/

• Implementing SLOs

Service level objectives (SLOs) specify a target level for the reliability of your service.

Why SREs Need SLOs

https://landing.google.com/sre/workbook/chapters/implementing-slos/

• Service Level Terminology

Indicators

An SLI is a service level indicator—a carefully defined quantitative measure of some aspect of the level of service that is provided.

Most services consider request latency—how long it takes to return a response to a request—as a key SLI. Other common SLIs include the error rate, often expressed as a fraction of all requests received, and system throughput, typically measured in requests per second. 

Another kind of SLI important to SREs is availability, or the fraction of the time that a service is usable. It is often defined in terms of the fraction of well-formed requests that succeed, sometimes called yield. (Durability—the likelihood that data will be retained over a long period of time—is equally important for data storage systems.)

For example, availabilities of 99% and 99.999% can be referred to as "2 nines" and "5 nines" availability, respectively, and the current published target for Google Compute Engine availability is “three and a half nines”—99.95% availability.

Objectives

An SLO is a service level objective: a target value or range of values for a service level that is measured by an SLI.

A natural structure for SLOs is thus SLI ≤ target, or lower bound ≤ SLI ≤ upper bound. For example, we might decide that we will return Shakespeare search results "quickly," adopting an SLO that our average search request latency should be less than 100 milliseconds.

Choosing an appropriate SLO is complex. To begin with, you don’t always get to choose its value! For incoming HTTP requests from the outside world to your service, the queries per second (QPS) metric is essentially determined by the desires of your users, and you can’t really set an SLO for that.

Agreements

Finally, SLAs are service level agreements: an explicit or implicit contract with your users that includes consequences of meeting (or missing) the SLOs they contain. The consequences are most easily recognized when they are financial—a rebate or a penalty—but they can take other forms. An easy way to tell the difference between an SLO and an SLA is to ask "what happens if the SLOs aren’t met?

SRE doesn’t typically get involved in constructing SLAs, because SLAs are closely tied to business and product decisions. SRE does, however, get involved in helping to avoid triggering the consequences of missed SLOs. They can also help to define the SLIs: there obviously needs to be an objective way to measure the SLOs in the agreement, or disagreements will arise.

Whether or not a particular service has an SLA, it’s valuable to define SLIs and SLOs and use them to manage the service.

SREs’ core responsibilities aren’t merely to automate “all the things” and hold the pager. Their day-to-day tasks and projects are driven by SLOs: ensuring that SLOs are defended in the short term and that they can be maintained in the medium to long term.SLOs are a tool to help determine what engineering work to prioritize. For example, consider the engineering tradeoffs for two reliability projects: automating rollbacks and moving to a replicated data store. By calculating the estimated impact on our error budget, we can determine which project is most beneficial to our users

https://landing.google.com/sre/sre-book/chapters/service-level-objectives/

network simulation

The Simulator tries to duplicate the behavior of the device.
The Emulator tries to duplicate the inner workings of the device.


Emulation is the replacement of a real world device with an model at a well defined interface for the purposes of allowing controlled responses from the emulated real world device. The emulation is "complete" if all the interfaces are present, and the resulting observed behavior matches that of the real world device.

Simulation is the use of modeling to create a controllable, representative stand in for a complex system. Simulations are, by definition, always incomplete.

An emulation is a system that behaves exactly like something else, and adheres to all of the rules of the system being emulated. It is effectively a complete replication of another system, right down to being binary compatible with the emulated system's inputs and outputs, but operating in a different environment to the environment of the original emulated system. The rules are fixed, and cannot be changed, or the system fails.

A simulation is a system that behaves similar to something else, but is implemented in an entirely different way. It provides the basic behaviour of a system, but may not necessarily adhere to all of the rules of the system being simulated. It is there to give you an idea about how something works.

https://stackoverflow.com/questions/2174638/whats-the-difference-between-emulation-and-simulation

• GNS3 2.1 Install and configuration on Windows 10 (Part 4): Basic GNS3 Network (your first network)

VPCS configuration must be saved in order to reopen the same project.

• Cisco Packet Tracer is a powerful network simulation program that allows students to experiment with network behavior and ask “what if” questions. As an integral part of the Networking Academy comprehensive learning experience, Packet Tracer provides simulation, visualization, authoring, assessment, and collaboration capabilities and facilitates the teaching and learning of complex technology concepts. 

http://www.cisco.com/web/learning/netacad/course_catalog/PacketTracer.html

• Images designed for running inside GNS3

The target is not to simulate the deployment of container infrastructure in production but use containers as light virtual machine replacing heavy qemu instance or VPCS when you want to use tools like telnet, nmap
It’s also not designed to control a docker cluster for production or development.  If you want to simulate real life containers infrastructure you need to deploy an OS on qemu and start container on it.
https://docs.gns3.com/1KGkv1Vm5EgeDusk1qS1svacpuQ1ZUQSVK3XqJ01WKGc/index.html

• It allows enterprises, e-learning providers/centers, individuals and group collaborators to create virtual proof of concepts, solutions and training environments.

http://www.eve-ng.com/

• UNetLabv2

labs distributed between dozens of physical or virtual nodes;
unlimited running labs for each user;
support for Ansible/NAPALM/… automation tools
http://www.routereflector.com/unetlab/#main

• Mininet creates a realistic virtual network, running real kernel, switch and application code, on a single machine (VM, cloud or native)

Mininet is also a great way to develop, share, and experiment with OpenFlow and Software-Defined Networking systems.
http://mininet.org/

SDN, OpenFlow and Mininet

• Mininet: Rapid Prototyping for Software Defined Networks

What is Mininet?
Mininet emulates a complete network of hosts, links, and switches on a single machine.
Mininet is useful for interactive development, testing, and demos, especially those using OpenFlow and SDN. OpenFlow-based network controllers prototyped in Mininet can usually be transferred to hardware with minimal changes for full line-rate execution.

How does it work?
Mininet creates virtual networks using process-based virtualization and network namespaces - features that are available in recent Linux kernels. In Mininet, hosts are emulated as bash processes running in a network namespace, so any code that would normally run on a Linux server (like a web server or client program) should run just fine within a Mininet "Host". The Mininet "Host" will have its own private network interface and can only see its own processes. Switches in Mininet are software-based switches like Open vSwitch or the OpenFlow reference switch. Links are virtual ethernet pairs, which live in the Linux kernel and connect our emulated switches to emulated hosts (processes).

https://github.com/mininet/mininet

• PacketCreator

PacketCreator is provided to help network administrators to test their network, by using ARP cache poisoning and generate ICMP packets to send over the LAN.
The main idea is to allow administrators using this kind of software to secure their network to several known attacks.
This tool can also redirect switched based networks traffic (dumping and forwarding are also provided).
It is possible to create a custom-made IP packet in which you can edit IP header or every ICMP field
https://www.softpedia.com/get/Network-Tools/Network-Testing/PacketCreator.shtml

• Nemesis

Nemesis is a command-line network packet crafting and injection utility for UNIX-like and Windows systems. Nemesis is well suited for testing Network Intrusion Detection Systems, firewalls, IP stacks and a variety of other tasks. As a command-line driven utility, Nemesis is perfect for automation and scripting.
http://nemesis.sourceforge.net/

• Nemesis Tutorial

In this tutorial, I show how to use nemesis to arp poison a windows 7 box.  Nemesis is a command-line packet injection too
http://pbnetworks.net/?cmd=bbs&id=41

• ettercap

For arp cache poisoning to take place, the attacker needs to be in the same network segment as the systems under attack. The first step is to obtain a list of IP addresses and their associated MAC addresses
https://ettercap.github.io/ettercap/

• dsniff

dsniff is a collection of tools for network auditing and penetration testing. dsniff, filesnarf, mailsnarf, msgsnarf, urlsnarf, and webspy passively monitor a network for interesting data (passwords, e-mail, files, etc.). arpspoof, dnsspoof, and macof facilitate the interception of network traffic normally unavailable to an attacker (e.g, due to layer-2 switching). sshmitm and webmitm implement active monkey-in-the-middle attacks against redirected SSH and HTTPS sessions by exploiting weak bindings in ad-hoc PKI.
http://monkey.org/~dugsong/dsniff/

• Arpwatch

A good defense against these techniques is to provide port security integrated into your switches and to run arpwatch to monitor address resolution protocol traffic on your network.
Arpwatch keeps track  for  ethernet/ip  address  pairings.  It  logs into syslog activity  and reports certain changes via email
http://linuxcommand.org/man_pages/arpwatch8.html

backbone networks

backbone network connects multiple LANs

In backbone network station/endpoint/server/pc is not directly connected to backbone

categories:

bus backbone

star backbone

connecting remote LANs

backbone handles a total load higher than one of the LANs backbone connecting

A backbone is a part of computer network that interconnects various pieces of network, providing a path for the exchange of information between different LANs or subnetworks. A backbone can tie together diverse networks in the same building, in different buildings in a campus environment, or over wide areas. Normally, the backbone's capacity is greater than the networks connected to it.

A large corporation that has many locations may have a backbone network that ties all of the locations together, for example, if a server cluster needs to be accessed by different departments of a company that are located at different geographical locations. The pieces of the network connections (for example: ethernet, wireless) that bring these departments together is often mentioned as network backbone. Network congestion is often taken into consideration while designing backbones.

A distributed backbone is a backbone network that consists of a number of connectivity devices connected to a series of central connectivity devices, such as hubs, switches, or routers, in a hierarchy. 

A collapsed backbone (inverted backbone, backbone-in-a-box) is a type of backbone network architecture

In the case of a collapsed or inverted backbone, each hub provides a link back to a central location to be connected to a backbone-in-a-box. That box can be a switch or a router. The topology and architecture of a collapsed backbone is a star or a rooted tree.

concepts & protocols

OSPF

BGP,MBGP

IS-IS

QoS

VRRP

MPLS:LDP,RSVP

L3 VPN

VRF /context routing

Policy based routing

Aggregation

Switching

VLAN

Trunk

802.1q VLAN tagging

STP/RSTP, Span Tree Protocol/Rapid Span Tree Protocol

EAPS, ethernet automatic protocol switching

L2 QoS

Stacking, when you have more than one switch

VRRP

Dynamic routing protocols

Link aggregation

Security

stateful inspection concept

zone/VR 

VSYS

dynamic routing protocols; OSPF,BGP

IPSEC(VPN)

NSRP(netscreen redundancy protocol)

screening options

SecDevOps / DevSecOps

• What is DevSecOps?

DevSecOps enables integration of security testing earlier in the software development lifecycle (SDLC). This is commonly referred to as “shifting security left” or “shift left.” DevSecOps enables seamless application security earlier in the software development lifecycle, rather than at the end when vulnerability findings requiring mitigation are more difficult and costly to implement.

DevSecOps is an extension of DevOps, and is sometimes referred to as Secure DevOps.

DevSecOps requires planning application and infrastructure security from the start

Benefits of DevSecOps

DevSecOps enables enhanced automation throughout the software delivery pipeline to eliminate coding mistakes and ultimately reduce breaches.

Teams that implement DevSecOps tools and processes to integrate security into their DevOps framework will be able to release secure software faster. Developers can test code for security and detect security flaws as code is written.

What are key components of DevSecOps

Application/API Inventory

    Automate the discovery, profiling, and continuous monitoring of the code across the portfolio. This may include production code in data centers, virtual environments, private clouds, public clouds, containers, serverless, and more. Use a combination of automated discovery and self-inventory tools

Custom Code Security

    Continuously monitor software for vulnerabilities throughout development, test, and operations. Deliver code frequently so vulnerabilities can be identified quickly with each code update

Static Application Security Testing (SAST) scans the application source files, accurately identifies the root cause and helps remediate the underlying security flaws.

Dynamic Application Security Testing (DAST) simulates controlled attacks on a running web application or service to identify exploitable vulnerabilities in a running environment.

Interactive Application Security Testing (IAST) provides a deep scan by instrumenting the application using agents and sensors to continuously analyze the application, its infrastructure, dependencies, dataflow, as well as all the code.

Open Source Security

    Open source software (OSS) often times includes security vulnerabilities, so a complete security approach includes a solution that tracks OSS libraries, and reports vulnerabilities and license violations.

    Software Composition Analysis (SCA) automates the visibility into open source software (OSS) for the purpose of risk management, security and license compliance. 

Runtime Prevention 

Protect applications in production – new vulnerabilities may be discovered, or legacy applications may not be in development. 

Runtime Application Self-Protection (RASP) instruments applications, directly measures attacks from the inside, and prevents exploits from within.

Compliance monitoring

    Enable audit readiness and a constant state of compliance for GDPR, CCPA, PCI, etc.

Making DevSecOps work for you

Step 1: Build Security into Software Requirements

Step 2: Test Early, Often and Fast

Step 3: Leverage Integrations to Make Application Security a Natural Part of the Lifecycle

Step 4: Automate Security as Part of the Development and Testing Processes

Step 5: Monitor and Protect Once Released

Automation in CI/CD pipelines

    SAST with Fortify Static Code Analyzer

    DAST with Fortify WebInspect

    RASP with Fortify Application Defender

    Software Composition Analysis (SCA) / Open Source Security (OSS) with Sonatype and Fortify

https://www.microfocus.com/en-us/what-is/devsecops

• DevOps isn’t just about development and operations teams. If you want to take full advantage of the agility and responsiveness of a DevOps approach, IT security must also play an integrated role in the full life cycle of your apps.

In the past, the role of security was isolated to a specific team in the final stage of development.
DevSecOps means thinking about application and infrastructure security from the start. It also means automating some security gates to keep the DevOps workflow from slowing down. Selecting the right tools to continuously integrate security can help meet your security goals, but effective DevOps security requires more than new tools—it builds on the cultural changes of DevOps to integrate the work of security teams sooner rather than later.

it has always been ideal to include security as an integral part of the entire app life cycle
DevSecOps is about built-in security, not security that functions as a perimeter around apps and data.

In part, DevSecOps highlights the need to invite security teams at the outset of DevOps initiatives to build in information security and set a plan for security automation.
It also underscores the need to help developers code with security in mind, a process that involves security teams sharing visibility, feedback, and insights on known threats.

For starters, a good DevSecOps strategy is to determine risk tolerance and conduct a risk/benefit analysis.
Automating repeated tasks is key to DevSecOps, since running manual security checks in the pipeline can be time intensive.

DevOps security is built for containers and microservices


DevSecOps means building security into app development from end to end.
DevOps teams should automate security to protect the overall environment and data, as well as the continuous integration/continuous delivery process—a goal that will likely include the security of microservices in containers.

Environment and data security:
Standardize and automate the environment.
Centralize user identity and access control capabilities.
Isolate containers running microservices from each other and the network.
Encrypt data between apps and services.
Introduce secure API gateways.

CI/CD process security:
Integrate security scanners for containers.
Automate security testing in the CI process.
Add automated tests for security capabilities into the acceptance test process.
Automate security updates, such as patches for known vulnerabilities.
Automate system and service configuration management capabilities.

https://www.redhat.com/en/topics/devops/what-is-devsecops

• DevSecOps is the philosophy of integrating security practices within the DevOps process. 

DevSecOps involves creating a ‘Security as Code’ culture with ongoing, flexible collaboration between release engineers and security teams.
In DevSecOps, two seemingly opposing goals —“speed of delivery” and “secure code”—are merged into one streamlined process
In alignment with lean practices in agile, security testing is done in iterations without slowing down delivery cycles. Critical security issues are dealt with as they become apparent, not after a threat or compromise has occurred.

Security protocols that are baked into the development process rather than added as a “layer on top” allows DevOps and security professionals to harness the power of agile methodologies.
As more organizations rely on cloud applications to keep operations up and running, security efforts independent of those performed by AWS are crucial to prevent costly downtimes.

DevSecOps vs. Rugged DevOps
Adding the term “rugged” to DevOps means adding increased trust, transparency, and a clearer understanding of probable risks. It is an accelerated approach where security parameters are put into practice at the start of the project and penetration tests applied throughout the development cycle.
In a DevSecOps environment, automated testing is performed throughout the development cycle. Ruggedizing the process means making security a higher priority. This includes incremental safety improvements in the continuous delivery pipeline (AWS or other), regular threat assessment using security games, and adding security testing to automated processes.

A cultural and technical shift towards a DevSecOps approach helps enterprises address security threats more effectively, in real-time

six important components of a DevSecOps approach
    Code analysis – deliver code in small chunks so vulnerabilities can be identified quickly.
    Change management – increase speed and efficiency by allowing anyone to submit changes, then determine whether the change is good or bad.
    Compliance monitoring – be ready for an audit at any time (which means being in a constant state of compliance, including gathering evidence of GDPR compliance, PCI compliance, etc.).
    Threat investigation – identify potential emerging threats with each code update and be able to respond quickly.
    Vulnerability assessment – identify new vulnerabilities with code analysis, then analyze how quickly they are being responded to and patched.
    Security training – train software and IT engineers with guidelines for set routines.


https://www.sumologic.com/devops/devsecops-rugged-devops/

• Security Assurance Requirements for Linux Application Container Deployments

https://nvlpubs.nist.gov/nistpubs/ir/2017/NIST.IR.8176.pdf

• An Unlikely Union: DevOps and Audit

https://itrevolution.com/book/devops-and-audit/

• Secure coding is a set of technologies and best practices for making software as secure and stable as possible. It encompasses everything from encryption, certificates, and federated identity to recommendations for moving sensitive data, accessing a file system, and managing memory.

The Fedora Project's Defensive Coding Guide provides guidelines for improving software security through secure coding
https://developers.redhat.com/topics/secure-coding/

• Open Source Identity and Access Management For Modern Applications and Services

User Federation, Identity Brokering and Social Login.
Add authentication to applications and secure services
https://www.keycloak.org/

• Is it DevSecOps or SecDevOps?

What Security Looks Like With SecDevOps
Security controls, guidelines, coding standards, and policies must be integrated completely into the software development process. This is done by making security part of the process and pipeline from the beginning — "Sec" then "Dev" then "Ops." The security team (or perhaps an architecture or senior developer specialized in security) defines the necessary policies upfront for the team.

These policies might consist of secure coding standards, rules for avoiding insecure APIs and poor encryption, instructions for using static and dynamic analysis, and testing guidelines. The goal is to have the developers working towards more secure software as part of their daily routine and automation helps make this a reality.

Security is often left as add-on or a gating process before releasing a product, but it's difficult to fix security issues when a product is halfway out the door. Shifting security to the left, as in SecDevOps, is the key to success.
https://dzone.com/articles/is-it-devsecops-or-secdevops

• What is SecDevOps and why should you care?

What is SecDevOps?
SecDevOps (also known as DevSecOps and DevOpsSec) is the process of integrating secure development best practices and methodologies into development and deployment processes which DevOps makes possible.

SecDevOps — sometimes called “Rugged DevOps” or “security at speed” — as a set of best practices designed to help organizations implant secure coding deep in the heart of their DevOps development and deployment processes. …It seeks to embed security inside the development process as deeply as DevOps has done with operations.

1. Security as Code (SaC): which refers to the building of security into the tools that exist in the DevOps pipeline. This means automation over manual processes. It means the use of static analysis tools that check the portions of code that have changed, versus scanning the entire code base.

2. Infrastructure as Code (IaC): defines the set of DevOps tools used to setup and update infrastructure components. Examples include Ansible, Chef, and Puppet. …With IaC, if a system has a problem, it is disintegrated, and a new one (or two) are created to fill the spot.

How Can SecDevOps Be Implemented?

Tooling

    Automate security audits. Use scripts, static and dynamic analysis, composition analysis, and integration of testing within existing tools
    Detect security flaws as soon as possible. The sooner a security flaw can be detected, and the further away they are done from production (or the client’s computer), the better it is, and the cheaper it is to resolve
    Regularly break the build. Ensure that tools can spot and flag security flaws which result in broken builds, just like how failing tests already work.
    Have accurate audit report results. Ensure that reports of security flaws are accurate; otherwise, faith and trust will rapidly erode.
    Use composition analysis. Ensure that you know the security, reliability, and exposure of the packages that you’re building your software on, as well as when they need to be avoided or replaced. Use tools that automatically validate them.
    Focus on instrumentation. Ensure that infrastructure, not just code, can be verified as working and secure; and that it’s replaceable when it’s isn’t.
    Use real-time protection. Ensure that production applications are protected against vulnerabilities that weren’t caught earlier. No software is 100% secure. Avoid solutions that create alert fatigue, false positives or that aren’t integrated into the DevOps tools.
   
Processes

    Establish strong feedback loops. As with any successful process or organization, you need to engender the ability to provide reliable feedback, even if the information delivered isn’t encouraging, or what the team wants to hear.
    Perform regular code audits. Just as with any other code review, such as for quality and standards compliance, security also needs to be reviewed, assessed, and corrected as transparently — and as quickly — as possible.
    Benchmark and review your performance. Like reaching any goal, you have to know where you are and whether you’re improving or declining in the attainment of it. Make sure that you know how you’re doing, and where you still need to improve.
    Have documented procedures for dealing with problems. Eventually, problems do occur. Ensure that you’re equipped to deal with them when they do in an organized and standardized manner.

Culture

To build a SecDevOps culture, consider the following four points:

    Engender a culture of openness and continuous learning. Ensure as much transparency as possible. Ensure that everyone in the team knows what’s going on, and is constantly encouraged to learn more.
    Build strong feedback loops. Ensure that information is rapidly delivered.
    Employ and nurture security evangelists. Ensure that you have people within your teams whose role is to reinforce and to grow security awareness and a security culture.
    Grow autonomy in every team. Ensure your teams can make the relevant decisions necessary to improve consistently.

   

https://blog.sqreen.com/secdevops/

• SecDevOps: Injecting Security Into DevOps Processes

As Farwell explains, a big part of SecDevOps is about empowering developers to become security experts.

Tools for SecDevOps

Stethoscope: Created by Netflix and now open source, Stethoscope helps security teams manage “user-focused security.” Security teams at companies like New Relic use Stethoscope to offer self-service end-user security to DevOps teams, so those teams can manage their own devices.

Claire: SecDevOps engineers use this open source project from CoreOS to scan for vulnerabilities in Docker containers.

Suricata: This open source tool can help detect threats against your networks. It uses rules, a signature language, and scripting tools to inspect network traffic and threats in real time. New Relic security engineers use Suricata to inspect the network traffic running through their container and cloud-based environment.

https://blog.newrelic.com/technology/what-is-secdevops/

• DevSecOps vs. SecDevOps vs. DevOpsSec: Is there really a difference in these secure DevOps terms?

Whether it is DevSecOps vs. SecDevOps vs. DevOpsSec, is there really a difference in these Secure DevOps terms? In this article, we will discuss the differences between each of these, and explain why SecDevOps is actually the correct security approach.

DevSecOps: So what is DevSecOps? If we break this down, you can see that Dev leads the pack and is given the highest level of importance.
As a result, DevSecOps represents the scenario we are in today – a situation where security doesn’t get the prioritization it deserves.

DevOpsSec: What is DevOpsSec? If taken literally, this version places security at the end of the line, after discrete development, deployment, and operations activities.

SecDevOps:SecDevOps refers to the inclusion of security efforts and best practices into the continuous integration and continuous deployment pipeline. It also suggests that security requirements are taken into account before development to ensure security is included throughout the product lifecycle.
https://www.cspi.com/devsecops-vs-secdevops-blog/

• Playbook Example: Continuous Delivery and Rolling Upgrades

What is continuous delivery?
Continuous delivery (CD) means frequently delivering updates to your software application.

Continuous delivery end-to-end

Now that you have an automated way to deploy updates to your application, how do you tie it all together? A lot of organizations use a continuous integration tool like Jenkins or Atlassian Bamboo to tie the development, test, release, and deploy steps together. You may also want to use a tool like Gerrit to add a code review step to commits to either the application code itself, or to your Ansible playbooks, or both.

https://docs.ansible.com/ansible/latest/scenario_guides/guide_rolling_upgrade.html

• Integrating Security Into the Devops Process

Ansible and Palo Alto Networks for Automation and Orchestration
Palo Alto Networks has built a series of Ansible modules that can help integrate security seamlessly into the DevOps process.
Ansible modules for Palo Alto Networks can be used to configure the entire family of next- generation firewalls, both physical virtualized form-factors as well as Panorama. The Ansible modules communicate with the next-generation firewalls and Panorama using the Palo Alto Networks XML API.

Import and load the configuration of the next-generation firewalls across virtual or physical deployments and/or integrate deployment within your existing CI/CD pipeline.
https://www.ansible.com/integrations/networks/palo-alto

• 6 reasons to use Ansible for network automation

1. Agentless
The importance of an agentless architecture cannot be stressed enough when it comes to network automation, especially as it pertains to automating existing devices.

As standalone network devices like routers, switches, and firewalls continue to add support for APIs, software-defined networking (SDN) solutions are also emerging. The one common theme with SDN solutions is that they all offer a single point of integration and policy management, usually in the form of an SDN controller. This is true for solutions such as Cisco ACI, VMware NSX, Big Switch Big Cloud Fabric, and Juniper Contrail, as well as many of the other SDN offerings from companies such as Nuage, Plexxi, Plumgrid, Midokura, and Viptela. This even includes open source controllers such as OpenDaylight.

As most software-defined networks are deployed with a controller, nearly all controllers expose a modern REST API. And because Ansible has an agentless architecture, it makes it extremely simple to automate not only legacy devices that may not have an API, but also software-defined networking solutions via REST APIs, all without requiring any additional software (agents) on the endpoints

2. Free and Open Source Software (FOSS)

3. Extensible
Ansible was primarily built as an automation platform for deploying Linux applications, although it has expanded to Windows since the early days

4. Integrating into Existing DevOps Workflows
Ansible is used for application deployments within IT organizations. It’s used by operations teams that need to manage the deployment, monitoring, and management of various types of applications.

5. Idempotency
The term idempotency (pronounced item-potency) is used often in the world of software development, especially when working with REST APIs, as well as in the world of DevOps automation and configuration management frameworks, including Ansible. One of Ansible’s beliefs is that all Ansible modules (integrations) should be idempotent

This is unlike most traditional custom scripts and the copy and pasting of CLI commands into a terminal window. When the same command or script is executed repeatedly on the same system, errors are (sometimes) raised.

Another example is if you have a text file or a script that configures 10 VLANs, the same commands are then entered 10 times every time the script is run. If an idempotent Ansible module is used, the existing configuration is gathered first from the network device, and each new VLAN being configured is checked against the current configuration. Only if the new VLAN needs to be added (or changed—VLAN name, as an example) is a change or command actually pushed to the device.

6. Network-Wide and Ad Hoc Changes
One of the problems solved with configuration management tools is configuration drift (when a device’s desired configuration gradually drifts, or changes, over time due to manual change and/or having multiple disparate tools being used in an environment)—in fact, this is where tools like Puppet and Chef got started

Because Ansible is agentless, there is not a default push or pull to prevent configuration drift

https://www.oreilly.com/ideas/6-reasons-to-use-ansible-for-network-automation

• Automating Cloud Security with Ansible and Palo Alto Networks

The Ansible modules for PAN-OS, our security operating system, allow our customers to embed security into the application development lifecycle, eliminating the bottleneck that change control security best practices can introduce.
In the public cloud, these collaboration efforts have become invaluable to our customers as they adopt more rapid and iterative application development methodologies (i.e., DevOps, CI/CD) on AWS, Azure and Google Cloud.
Using Ansible modules to deploy and configure Palo Alto Networks VM-Series firewalls on AWS, Azure and Google Cloud
https://blog.paloaltonetworks.com/2018/04/automating-cloud-security-ansible-palo-alto-networks/

• What is BizDevOps?

In fact, some DevOps teams could already call themselves BizDevOps. These teams have business domain owners participating from the start, they use Kanban and scrum (plus more), and are leveraging past learnings to eliminate waste

The team makeup: Business team and/or application owner, development team members, and operation team members form a single team; focus could be a product your company delivers, or a business process, business component, or business service.

Integrated value and flow of processes: You have integrated business leaders, developers, and operations folks who all are working on a streamlined flow from your company’s strategy to the deployment and ongoing operations of the product, service, or component. You understand the value stream or you are working on, and you understand the customer journey end-to-end, including all key value chain components.

Existence of key performance metrics: You have jointly established key performance indicators, which include customer-centric metrics – e.g., user behavior and feature adoption to DevOps metrics such as time-to-business-impact and speed of remediation.

Automation toolchain: You are leveraging a variety of automation platforms that allow you to collaborate and automate across the different process and data items.

https://enterprisersproject.com/article/2019/9/devops-what-is-bizdevops

• It’s proven that DevOps increases the ability of an organization to deliver services at high velocity.

But what if the same tools and practices that accelerate feature delivery can also be used to generate greater business value?

That’s where BizDevOps comes in.

BizDevOps integrates feedback from the business side of an organization into its delivery cycles, effectively ensuring features released through DevOps cycles are built specifically to serve business objectives. A streamlined workflow is created from business strategy to deployment, allowing DevOps metrics to become aligned with high-level business KPIs.

Just as DevOps overcomes silos separating development from operations, BizDevOps overcomes silos separating DevOps from the rest of the business.

BizDevOps is accomplished by encouraging the business team to work directly with product owners, developers, and operators to set priorities for sprints and backlogs.

Define metrics that measure business value, and make sure your deployment and release strategies take traditional business concerns, such as geography, community, and other internal and external factors, into account.

https://medium.com/@CloudOps_/what-is-bizdevops-642806ab6c60