fakecineaste : Testing

How to use software metrics to manage software developers
Metrics-Based software management

Try to be a mentor,a model or a father model(this is necessary for verbal communication)
- common software metrics (measurements)

Software metric

[

Connascent

Cyclomatic

tool,that

are derived

the source code of a program has been tested

strongly-related

given

tend to be

is said

highly-cohesive

reuse

is increased

complexity is kept

Connascent

connascent

be modified

in order

is usually contrasted

Cyclomatic

is used

indicate

evaluate

single

is calculated

These metrics

are therefore computed

The total path length for the entire program could be deemed

text of the program

SLOC is typically used

be required

as well as

is produced

lifecycle

etc

that is

the process of

executable

type of

been mentioned

Software metric

In general the

be computed

’s going to

low cost

Effectively inspect

Obtain

–

White box testing also named as clear box testing, transparent testing, glass box testing and structural testing.

It is a method in which the internal structure of application is tested.

Black box testing is a method that tests for the functionality of an application.
Black box testing sometimes called Behavioral Testing

Recovery testing is a method for testing how well a software can recover from crashes.
Security testing ensures that the software protects the data and performs its all functions.
Stress testing determines the robustness of software.

The main objecting of testing is to make the software error free, to uncover errors

Introduction to Software Metrics

Software metrics are used to give programmers feedback about their program. Metrics can be used as guidelines for when to refactor. They provide a way to measure the progress of code during development. In this tutorial, we are going to use Metrics, an open-source Eclipse plug-in, to gather metrics on a project.

http://agile.csc.ncsu.edu/SEMaterials/tutorials/metrics/

tools to improve software quality

Goals of a software development team

Automated Testing

Continuous Integration

Code Quality Metrics

Technical Documentation

Traditional development processes

– Room for improvement?

? Poorly tested applications?

? Applications that are difficult and expensive to change?

? Lots of bugs at delivery time?

? Inconsistent coding standards and programming habits?

? Code that is hard to maintain and to update

? Technical documentation out of date?

How can we improve?

– Enforce good testing practices

– Monitor developer test quality and coverage

– Automate the build process

– Monitor and review code quality metrics

– Automatic technical documentation

Several types of developer tests

– Unit Tests

– Integration Tests

– Graphical User Interface (Web) Tests

All can be automated to varying degrees

Unit Tests

– Unit tests can help you

? Ensure that code behaves as expected

? Make your code more flexible and easier to maintain

? Detect regressions

? Document your code

What is regression?

A software regression is a software bug which makes a feature stop functioning as intended after a certain event (for example, a system upgrade, system patching or a change to daylight saving time).

A software performance regression is a situation where the software still functions correctly, but performs slowly or uses more memory when compared to previous versions.

Regressions are often caused by encompassed bug fixes included in software patches.

One approach to avoiding this kind of problem is regression testing.

A properly designed test plan prevents this possibility before releasing any software.

Automated testing and well-written test cases can reduce the likelihood of a regression.

A software regression can be of one of three types:

Local – a change introduces a new bug in the changed module or component.

Remote – a change in one part of the software breaks functionality in another module or component.

Unmasked – a change unmasks an already existing bug that had no effect before the change.

http://en.wikipedia.org/wiki/Software_regression

Traditional testing vs Unit testing

Traditional developer testing

Run against the entire application

Run manually in a debugger

Testing is ad-hoc and not reproduc

Testing needs human intervention

Unit testing

Run against classes or small components

Tests can be run automatically

Tests can be reused

Testing can be automated

? Unit Tests

– The costs of writing unit tests

? More code to write and maintain

? More time required to write the code, initially

Unit Tests

– The benefits of writing unit tests

? More reliable code with less bugs

? More flexible code

? Code that is easier to maintain

? Automatic regression tests

A good unit test should:

? Test individual classes or small components

? Test code in isolation

? Run quickly

? Leave the system in a predictable state

? Be reproducible

? Be automated

Unit Testing tools

– Junit 4

? A newer, more modern version of JUnit

– TestNG

? A more recent and very innovative unit testing library

? Less widely used than JUnit

Testing web interfaces

– Often neglected by developers and left to the testers

– Traditionally difficult to automate

– Involves much human judgement

Automate web interface testing

Automatic smoke tests

Automatic regression tests

Web Interface Testing technologies

– Using Mock Objects

? Spring mocks,mockito,Spring MVC, StrutsTestCase, JSFUnit

? Good for the controller layer

? Doesn't test the HTML screens themselves

– Writing tests to run against a web application

? HTTPUnit, Cactus, Jmeter, TestMaker,HTMLUnit,JWebUnit

– Run tests from within a real web browser

? Selenium,WebDriver

Testing web applications with Selenium

– Selenium is a browser-based testing tool

– You can

? Record test scripts in a browser

? Replay them manually or automatically

-Firefox Selenium IDE

Test Coverage

– Test Coverage indicates how much application code is executed by your unit tests

– It is especially useful for identifying code that has not been tested.

Test Coverage Tools

– Automated Test Coverage

? Integrated into the build process

? Runs for every build

? Team-wide reporting

? Tools like Cobertura

Test coverage in your IDE

? Faster feedback for developers

? ECLEmma - Test Coverage in Eclipse

? Crap4j – also provides test coverage metrics

Why use EclEmma/Crap4j and Cobertura

– EclEmma and Crap4j allows IDE integration

? Fast feedback for the developer

? More convenient for the developer than using an HTML report

– Cobertura allows project-level coverage reporting

? Project-wide coverage statistics

? Results can be published and reviewed

? Can be used to enforce minimum coverage levels

The principal components of a Continuous Integration service:

-automated build process

ant,maven,makefile

-automated test process

junit,testng

-source code repository

csv,subversion,git

-continuous build tool

cruisecontrol,jenkins,hudson,continuum,luntbuild

The “traditional” software process involves:

? Coding

? Ad-hoc testing

? Commit changes shortly before the start of the testing phase

? Difficult integration process

ad-hoc=not planned,but arranged or done when necessary

Continuous Integration:

? Smoother integration

? Automatic regression testing

? Regular working releases

? Earlier functional testing

? Faster and easier bug fixes

? Better visibility

Continuous Integration tools:

Open Source tools

? CruiseControl, Hudson,Jenkins, Continuum, LuntBuild

– Commercial Tools

? TeamCity, Bamboo, Pulse,

Quality Metrics Tools

-Several complementary tools

• Checkstyle – enforce coding standards,naming conventions,javadocs

• PMD – best practices

• FindBugs – potential bugs

• Crap4j – overly complex and poorly tested classes

Checkstyle - Enforce coding standards

-Eclipse plugin

-Hudson reports

-Maven reports

PMD – Best practices

-Empty try/catch/finally blocks

-Incorrect null pointer checks

-Excessive method length or complexity

-Some overlap with Checkstyle

-Eclipse plugin

-Hudson reports

-Maven reports

FindBugs – Potential defects

-Potential NullPointerExceptions

-Infinite loops

-Eclipse plugin

-Hudson reports

-Maven reports

Crap4j – Complex and untested code

-Uses code coverage and code complexity metrics

-Eclipse plugin

Automated Documentation

-Human-written documentation – use a project Wiki

-Automatically-generated UML/Javadoc

? UmlGraph

-Automatically-generated database models

? SchemaSpy

SoftwareQualityNZ.pdf john.smart@wakaleo.com

the acronym DTAP—Development, Testing, Acceptance, and Production.

One thing that has changed recently, though, is that these systems no longer have to mean separate hardware. It is acceptable to have one or more of these systems set up as virtual servers.

https://www.phparch.com/2009/07/professional-programming-dtap-%E2%80%93-part-1-what-is-dtap/

What Is Behavior Driven Testing?

Behavior Driven Testing (BDT) is an uncommon term in software testing/development compared to Behavior Driven Development (BDD). Tests in behavior driven testing are most often focused on the behavior of users rather than the technical functions of the software
https://dzone.com/articles/behavior-driven-testing-in-automated-testing-2

Analogous to test-driven development, Acceptance Test Driven Development (ATDD) involves team members with different perspectives (customer, development, testing) collaborating to write acceptance tests in advance of implementing the corresponding functionality. The collaborative discussions that occur to generate the acceptance test is often referred to as the three amigos, representing the three perspectives of customer (what problem are we trying to solve?), development (how might we solve this problem?), and testing (what about...).

https://www.agilealliance.org/glossary/atdd/

Testing

In order to prove that a piece of software works, the software must be tested to determine if the requirements of the application are met. There are several different types of tests used throughout the development process.

The two main types of testing are white box and black box testing.

White box test cases are used to test specific paths through the code.
Black box tests treat the program like a black box; what happens in the program is invisible and unimportant to the user.
Black box test cases only look at specific inputs and outputs of an application.

There are several different levels of testing that are done throughout the software development process.

Acceptance Test: Testing conducted by a customer to verify that the system meets the acceptance criteria of the requested application.(Black Box)
Integration Test: Tests the interaction of small modules of a software application.(White or Black Box)
Unit Test: Tests a small unit (i.e. a class) of a software application, separate from other units of the application. (White Box)
Regression Test: Tests new functionality in a program. Regression testing is done by running all of the previous unit tests written for a program, if they all pass, then the new functionality is added to the code base. (White Box)
Functional and System Test: Verifies that the entire software system satisfies the requirements. (Black Box)
Beta Test: Ad-hoc, third party testing. (Black Box)

Regression testing

Regression testing is any type of software testing that seeks to uncover new software bugs, or regressions, in existing functional and non-functional areas of a system after changes, such as enhancements, patches or configuration changes, have been made to them.
The intent of regression testing is to ensure that a change such as those mentioned above has not introduced new faults.
One of the main reasons for regression testing is to determine whether a change in one part of the software affects other parts of the software
http://en.wikipedia.org/wiki/Regression_testing

Unit Testing

Describes the process of testing modular sections of an application.
Integration Testing
Explains the process of testing the combination of two or more sections of an application.
Regression Testing
Describes the process of retesting an application following implementation changes.
http://msdn.microsoft.com/en-us/library/aa292484%28v=vs.71%29.aspx

There are several times during the software development process that testing can be done.

In the waterfall software process, testing is done after all code has been developed and before release of the product.
The spiral model has a quadrant devoted to testing. Testing is done several times as the development "spirals in."
Agile software methods advocate test driven development (TDD), which means that test cases are written before any coding is done.

Verification and validation of a software application are an important result of software testing.

Verification and validation can also be done by code inspections and reviews.
Verification determines if the product developed during a certain phase of development meets the requirements determined in the previous phase of development.
Validation determines if a final piece of software meets the customer requirements.

http://openseminar.org/se/modules/7/index/screen.do

Differences between Verification and Validation

Verification is the process of checking that a software achieves its goal without any bugs.

Verification means Are we building the product right?

It includes checking documents, design, codes and programs

Verification is static testing.

It does not include the execution of the code.

Methods used in verification are reviews, walkthroughs, inspections and desk-checking.

It checks whether the software conforms to specifications or not.

It can find the bugs in the early stage of the development.

The goal of verification is application and software architecture and specification.

Quality assurance team does verification.

It comes before validation

It consists of checking of documents/files and is performed by human.

Validation is the process of checking whether the software product is up to the mark or in other words product has high level requirements.

Validation means Are we building the right product?

It includes testing and validating the actual product.

Validation is the dynamic testing.

It includes the execution of the code.

Methods used in validation are Black Box Testing, White Box Testing and non-functional testing.

It checks whether the software meets the requirements and expectations of a customer or not.

It can only find the bugs that could not be found by the verification process.

The goal of validation is an actual product.

Validation is executed on software code with the help of testing team.

It comes after verification.

It consists of execution of program and is performed by computer.

https://www.geeksforgeeks.org/differences-between-verification-and-validation/

Verification and validation are independent procedures that are used together for checking that a product, service, or system meets requirements and specifications and that it fulfills its intended purpose

"Validation. The assurance that a product, service, or system meets the needs of the customer and other identified stakeholders. It often involves acceptance and suitability with external customers. Contrast with verification."

"Verification. The evaluation of whether or not a product, service, or system complies with a regulation, requirement, specification, or imposed condition. It is often an internal process. Contrast with validation."

https://en.wikipedia.org/wiki/Verification_and_validation

What is the difference between black- and white-box testing?

• black-box (procedural) test: Written without knowledge of
how the class under test is implemented.
– focuses on input/output of each component or call

• white-box (structural) test: Written with knowledge of the
implementation of the code under test.
– focuses on internal states of objects and code
– focuses on trying to cover all code paths/statements
– requires internal knowledge of the component to craft input
• example: knowing that the internal data structure for a spreadsheet
uses 256 rows/columns, test with 255 or 257
https://docs.google.com/viewer?a=v&q=cache:dRmPa0PYWw4J:www.cs.washington.edu/education/courses/cse403/12wi/lectures/15-blackwhiteboxtesting.ppt+&hl=en&pid=bl&srcid=ADGEESia6XVFcpnj-cC2nAKGG54P7a4EVxIxB4aH2U_IRugVT0jZ_wQHn1ad1DOCzFVte5WYDPXyOz8ApwixTqVMDlH3y9iDq49rXBUQNh3LdEq9pjCc0y7XnTD18NUwhnKaqI7_R1JX&sig=AHIEtbQ4Mhc6f8GCaJdpyCj02iP34GxilA

Penetration test(pen-test)

A penetration test, occasionally pentest, is a method of evaluating the security of a computer system or network by simulating an attack from malicious outsiders (who do not have an authorized means of accessing the organization's systems) and malicious insiders (who have some level of authorized access).

http://en.wikipedia.org/wiki/Penetration_test

Mock object

In object-oriented programming, mock objects are simulated objects that mimic the behavior of real objects in controlled ways.

A programmer typically creates a mock object to test the behavior of some other object, in much the same way that a car designer uses a crash test dummy to simulate the dynamic behavior of a human in vehicle impacts

Reasons for use

In a unit test, mock objects can simulate the behavior of complex, real (non-mock) objects and are therefore useful when a real object is impractical or impossible to incorporate into a unit test.

If an object has any of the following characteristics, it may be useful to use a mock object in its place:

supplies non-deterministic results (e.g., the current time or the current temperature);

has states that are difficult to create or reproduce (e.g., a network error);

is slow (e.g., a complete database, which would have to be initialized before the test);

does not yet exist or may change behavior;

would have to include information and methods exclusively for testing purposes (and not for its actual task).

For example, an alarm clock program which causes a bell to ring at a certain time might get the current time from the outside world. To test this, the test must wait until the alarm time to know whether it has rung the bell correctly. If a mock object is used in place of the real object, it can be programmed to provide the bell-ringing time (whether it is actually that time or not) so that the alarm clock program can be tested in isolation.

http://en.wikipedia.org/wiki/Mock_object

What is the objective of preparing the test plan?

The main objective of creating Test plan is:

Project scope
Roles and Responsibilities
Deadlines and deliverables.

What is the meaning of Code Inspection?
Code inspection allows the programmer to review their source code with a group who ask questions related to program logic, analyzing the code. It checks against some most common programming errors and verifies coding standards.

State main difference Between Quality Assurance and Quality Control?
QA is more planned and systematic method of monitoring the quality of the process. This process is followed to produce quality software and application.
Quality Control is related to the quality of the product. QC team never find the defects and suggests improvements.
What are Alpha and Beta Testing?
Alpha Testing helps to identify all possible bugs before releasing the product to everyday users or the public. Beta Testing is performed by users of the software application in a real environment.
What is the definition of ad hoc testing?
Adhoc testing is informal testing. It is performed without any planning and documentation. This kind of test is conducted randomly without any expected results.

https://career.guru99.com/sdet-interview-questions-answers/

Difference Between Quality Assurance(QA) and Testing

Quality Assurance is a set of methods and activities designed to ensure that the developed software corresponds to all the specifications, e.g., SRS, FRS, and BRS.
It is a planned strategy of the testing process evaluation aimed at the quality product yield.
QA works out ways to prevent possible bugs in the process of software development.
QA deals more with the management stuff: methods and techniques of development, project analysis, checklists
QA goes through the whole product life cycle (SDLC) and heads the process of software maintenance.

Software Testing is a way of exploring the system to check how it operates and find the possible defects. Various methods are used to test the product, locate bugs and check if they are fixed. Testing enables customers with the possibility to see if the developed product meets their expectations on its design, compatibility, functioning, etc.
Despite the fact that software testing works out ways to break the software (aiming to check the system), it is a complex and crucial process. Testing is product-oriented, time-limited and reports to QC process.
https://blog.qatestlab.com/2011/04/07/what-is-the-difference-between-qa-and-testing/

What is the Difference Between SRS, FRS and BRS

SRS – software requirement specification
FRS – functional requirement specification
BRS – business requirement specification

Software Specification describes a software system that should be developed. This type of specification holds functional and non-functional requirements as well as the use cases. This is the base for a contract with a client that contains details on how the software must run. Developers constantly use SRS in the process of a product/program development.
Functional Specification determines the functions that the product must perform. This document explains the behavior of the program components in the process of interaction with a user.
Business Specification points out the means on how to meet the requirements in business. Very often BRS depicts the expected features of a software or a product to be developed. Business analysts develop BRS according to the specifications of a definite product.

Use Cases: SRS describes the interaction between the created product and the end users. It is the reason why this specification type includes use cases. FRS, as well as BRS, has nothing in common with use cases.
Development: A system analyst is responsible for SRS creation, while developers – for FRS. BRS is a duty of a business analyst.
https://blog.qatestlab.com/2015/12/31/srs-frs-brs/

Quality Assurance vs Quality Control vs Testing

The relationship between QA, QC, and Testing has a hierarchical nature. At the top of the pyramid is QA. Its main aim is to plan and establish the processes of quality evaluation. QC specifies the implementation of QA processes. As a part of QC, Testing is a way to gather information about software quality in order to check the results of implemented QC processes.
Testing
Software Testing is a process of verifying different attributes of a system and aspects of its usage in order to make sure that the product operates as expected and that it does not perform any functions that it is not supposed to conduct. Testing is a part of QC.
http://qatestlab.com/resources/knowledge-center/quality-assurance-control/

PythonTestingToolsTaxonomy

Unit Testing Tools
Mock Testing Tools
Fuzz Testing Tools
According to Wikipedia, "fuzz testing" (or "fuzzing") is a software testing technique whose basic idea is to attach the inputs of a program to a source of random data ("fuzz").
Web Testing Tools
Acceptance/Business Logic Testing Tools
GUI Testing Tools
Source Code Checking Tools
Code Coverage Tools
Continuous Integration Tools
Automatic Test Runners
Test Fixtures
https://wiki.python.org/moin/PythonTestingToolsTaxonomy

Sonar

Sonar is an open platform to manage code quality. As such, it covers the 7 axes of code quality:
http://www.sonarsource.org/

Checkstyle

Checkstyle is a development tool to help programmers write Java code that adheres to a coding standard.
http://checkstyle.sourceforge.net/

PMD is a source code analyzer. It finds unused variables, empty catch blocks, unnecessary object creation, and so forth.
http://pmd.sourceforge.net/

Code Quality Tools Review for 2013: Sonar, Findbugs, PMD and Checkstyle

code quality tools fulfill a growing need, as our code bases become larger and more complex, and it’s important to try to automate your code checks as much as possible. They are pretty versatile and customizable, and typically they are integrated into your build process, but can also be run manually in a one-off fashion.
http://zeroturnaround.com/labs/code-quality-tools-review-for-2013-sonar-findbugs-pmd-and-checkstyle/#!/

FindBugs

FindBugs, a program which uses static analysis to look for bugs in Java code.
http://findbugs.sourceforge.net/

Simian (Similarity Analyser) identifies duplication in Java, C#, C, C++, COBOL, Ruby, JSP, ASP, HTML, XML, Visual Basic, Groovy source code and even plain text files. In fact, simian can be used on any human readable files such as ini files, deployment descriptors, you name it.

http://www.harukizaemon.com/simian/

Doxygen a tool for writing software reference documentation. The documentation is written within code, and is thus relatively easy to keep up to date. Doxygen can cross reference documentation and code, so that the reader of a document can easily refer to the actual code.

Multivariate Testing and A/B testing

What Is Multivariate Testing?

[# Variations on Element A] X [# Variations on Element B]….. = [Total # Variations]

The main aim of performing Multivariate testing is to measure and determine the effectiveness of each variation combination on the final system.

After finalizing the variation combinations, testing to determine the most successful design is initiated once enough traffic is received by the site

A/B Testing

In A/B testing, two versions of the same webpage are put under test with an equal amount of webpage traffic. The version which gets a maximum number of conversion is the ultimate winner. This new version definitely increases the conversion rate.

https://www.softwaretestinghelp.com/multivariate-testing-and-ab-testing/

The solution is to use a concept from probability theory known as the multi-armed bandit (or MAB).Conceptually, the multi-armed bandit assumes you enter a casino. You are faced with a number of slot machines with different rules and payouts. Some machines will pay out small sums regularly, others may pay out large sums, but only rarely. You have limited resources, but you want to know how to maximise your winnings across the night. Importantly, before playing, you have no prior knowledge of the rules for each machine. If you had infinite time and money, you could easily solve this. But you only have a fixed time and limited money.

Epsilon-greedy method

The first MAB solution that Marktplaats looked at is called epsilon-greedy.Epsilon-greedy has an exploration phase where you quickly pick the most successful strategy. During this phase, people are shown a random recommender. Then there is an exploitation phase, where you show, say, 90% of users the best recommender and the other 10% are shown a random choice (where 10% = epsilon = 0.1). You can either go straight to the exploitation phase after one round, or you can be more nuanced and home in on the most successful recommender slowly.While this approach gave eBay double the revenue of the random A/B testing method, it still has drawbacks. Firstly, it still isn’t personalised. Secondly, it homes in on a different choice each time it is run.

Contextual bandits with Thompson sampling

Thompson sampling isn’t greedy. Instead, it uses a Bayesian approach. As Wikipedia says, it tries to choose the action that “maximizes the expected reward with respect to a randomly drawn belief”. Rather than randomly assigning users, it is able to use complex vectors of features when choosing which recommender to assign a given user. In eBay’s case, they chose the last five categories the user has visited and whether they are on a mobile or desktop.

One big problem with this approach is it takes twice as long to “unlearn” a choice than it does to learn.

https://www.codemotion.com/magazine/dev-hub/designer-cxo/multi-armed-bandits-a-better-way-to-a-b-test/

The Testing Pyramid

The Testing Pyramid has three classic layers:

Unit tests are at the bottom. Unit tests directly interact with product code, meaning they are “white box.” Typically, they exercise functions, methods, and classes. Unit tests should be short, sweet, and focused on one thing/variation. They should not have any external dependencies – mocks/monkey-patching should be used instead.

Integration tests are in the middle. Integration tests cover the point where two different things meet. They should be “black box” in that they interact with live instances of the product under test, not code. Service call tests (REST, SOAP, etc.) are examples of integration tests.

End-to-end tests are at the top. End-to-end tests cover a path through a system. They could arguably be defined as a multi-step integration test, and they should also be “black box.” Typically, they interact with the product like a real user. Web UI tests are examples of integration tests because they need the full stack beneath them.

The Testing Pyramid is triangular for a reason: there should be more tests at the bottom and fewer tests at the top. Why?

Distance from code. Ideally, tests should catch bugs as close to the root cause as possible. Unit tests are the first line of defense. Simple issues like formatting errors, calculation blunders, and null pointers are easy to identify with unit tests but much harder to identify with integration and end-to-end tests.

Execution time. Unit tests are very quick, but end-to-end tests are very slow. Consider the Rule of 1’s for Web apps: a unit test takes ~1 millisecond, a service test takes ~1 second, and a Web UI test takes ~1 minute. If test suites have hundreds to thousands of tests at the upper layers of the Testing Pyramid, then they could take hours to run

Development cost. Tests near the top of the Testing Pyramid are more challenging to write than ones near the bottom because they cover more stuff. They’re longer. They need more tools and packages (like Selenium WebDriver). They have more dependencies

Reliability. Black box tests are susceptible to race conditions and environmental failures, making them inherently more fragile. Recovery mechanisms take extra engineering.

The total cost of ownership increases when climbing the Testing Pyramid. When deciding the level at which to automate a test (and if to automate it at all), taking a risk-based strategy to push tests down the Pyramid is better than writing all tests at the top. Each proportionate layer mitigates risk at its optimal return-on-investment.

https://automationpanda.com/2018/08/01/the-testing-pyramid/

Test Pyramid: the key to good automated test strategy

The test pyramid is a tool to fix the problem of over-reliance on long-running UI tests.

The pyramid says that tests on the lower levels are cheaper to write and maintain, and quicker to run. Tests on the upper levels are more expensive to write and maintain, and slower to run. Therefore you should have lots of unit tests, some service tests, and very few UI tests.

In this situation the QA department has created an automated test suite, but the development team has not. It will be very long running and flakey because the development team has not helped build the suite or architect the application in a way that makes it easy to test. It is broken by the devs very regularly and they are relying on the QA department to fix it.

Both the hourglass and the ice cream cone is an indicator that there is a lack of collaboration and communication between the QA and Development departments, they are probably separated organizationally and even by location.

https://medium.com/@timothy.cochran/test-pyramid-the-key-to-good-automated-test-strategy-9f3d7e3c02d5

Just Say No to More End-to-End Tests

Unit Tests

Unit tests take a small piece of the product and test that piece in isolation. They tend to create that ideal feedback loop:

Unit tests are fast. We only need to build a small unit to test it, and the tests also tend to be rather small. In fact, one tenth of a second is considered slow for unit tests.

Unit tests are reliable. Simple systems and small units in general tend to suffer much less from flakiness. Furthermore, best practices for unit testing - in particular practices related to hermetic tests - will remove flakiness entirely.

Unit tests isolate failures. Even if a product contains millions of lines of code, if a unit test fails, you only need to search that small unit under test to find the bug.

Unit Tests vs. End-to-End Tests

With end-to-end tests, you have to wait: first for the entire product to be built, then for it to be deployed, and finally for all end-to-end tests to run. When the tests do run, flaky tests tend to be a fact of life. And even if a test finds a bug, that bug could be anywhere in the product.

Although end-to-end tests do a better job of simulating real user scenarios, this advantage quickly becomes outweighed by all the disadvantages of the end-to-end feedback loop:

Integration Tests

Unit tests do have one major disadvantage: even if the units work well in isolation, you do not know if they work well together. But even then, you do not necessarily need end-to-end tests. For that, you can use an integration test. An integration test takes a small group of units, often two units, and tests their behavior as a whole, verifying that they coherently work together

If two units do not integrate properly, why write an end-to-end test when you can write a much smaller, more focused integration test that will detect the same bug?

Testing Pyramid

Even with both unit tests and integration tests, you probably still will want a small number of end-to-end tests to verify the system as a whole. To find the right balance between all three test types, the best visual aid to use is the testing pyramid.

The bulk of your tests are unit tests at the bottom of the pyramid. As you move up the pyramid, your tests gets larger, but at the same time the number of tests (the width of your pyramid) gets smaller.

As a good first guess, Google often suggests a 70/20/10 split: 70% unit tests, 20% integration tests, and 10% end-to-end tests. The exact mix will be different for each team, but in general, it should retain that pyramid shape. Try to avoid these anti-patterns:

Inverted pyramid/ice cream cone. The team relies primarily on end-to-end tests, using few integration tests and even fewer unit tests.

Hourglass. The team starts with a lot of unit tests, then uses end-to-end tests where integration tests should be used. The hourglass has many unit tests at the bottom and many end-to-end tests at the top, but few integration tests in the middle.

https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html

The Hermetic test pattern

The Hermetic test pattern is the polar opposite of the Spaghetti pattern; it states that each test should be completely independent and self-sufficient. Any dependency on other tests or third-party services that cannot be controlled should be avoided at all costs.

https://subscription.packtpub.com/book/web_development/9781783982707/3/ch03lvl1sec24/the-hermetic-test-pattern

In computer science, test coverage is a measure used to describe the degree to which the source code of a program is executed when a particular test suite runs. A program with high test coverage, measured as a percentage, has had more of its source code executed during testing, which suggests it has a lower chance of containing undetected software bugs compared to a program with low test coverage

Basic coverage criteria

There are a number of coverage criteria, the main ones being:

Function coverage – has each function (or subroutine) in the program been called?

Statement coverage – has each statement in the program been executed?

Edge coverage – has every edge in the Control flow graph been executed?

Branch coverage – has each branch (also called DD-path) of each control structure (such as in if and case statements) been executed? For example, given an if statement, have both the true and false branches been executed? This is a subset of edge coverage.

Condition coverage (or predicate coverage) – has each Boolean sub-expression evaluated both to true and false?

https://en.wikipedia.org/wiki/Code_coverage

What is Code Coverage?

Code coverage is performed to verify the extent to which the code has been executed. Code coverage tools use static instrumentation in which statements monitoring code execution are inserted at necessary junctures in the code.

Code coverage scripts generate a report that details how much of the application code has been executed. This is a white-box testing technique

Why perform Code Coverage?

Code coverage is primarily performed at the unit testing level. Unit tests are created by developers, thus giving them the best vantage from which to decide what tests to include in unit testing. At this point, a number of questions arise:

Are there enough tests in the unit test suite?

Do more tests need to be added?

Code coverage answers these questions.

As development progresses, new features and fixes are added to the codebase. Obviously, the test code must be changed to stay updated with these changes. Testing standards established at the beginning of the project must also be maintained throughout subsequent release cycles. Code coverage can ensure that these standards are maintained so that only the optimal quality code is pushed to production.

A high percentage of code coverage results in lower chances of unidentified bugs. It is best to set a minimum percentage of code coverage that must be achieved before going to production. This helps reduce the chances of bugs being detected at a later stage of development.

Advantages of Code Coverage

Quantitative: Code coverage offers results in quantitative metrics which helps developers gauge the nature and health of their code.

Allows introduction of test cases: If already available test cases do not test the software extensively enough, one can introduce their own test cases to establish robust coverage.

Easy elimination of dead code and errors: Let’s say some parts of the entire codebase were not touched during code coverage, or there are sections of dead or useless code. Code coverage allows easy removal of such code, thus improving the efficiency of the entire code base.

What is Test Coverage?

Unlike code coverage, test coverage is a black-box testing technique. It monitors the number of tests that have been executed. Test cases are written to ensure maximum coverage of requirements outlined in multiple documents – FRS (Functional Requirements Specification), SRS (Software Requirements Specification), URS (User Requirement Specification), etc.

The test coverage report provides information about parts of the software where test coverage is being implemented. Essentially, it provides information about the tests executed on an application or website.

How To Perform Test Coverage?

For example, user-centric web apps prioritize UI/UX tests over functional tests. Conversely, financial apps will prioritize usability and security testing over all other tests.

Some of the test coverage mechanisms:

Unit Testing: Performed at a unit level or module level. Bugs at this level are widely different from issues encountered at the integration stage.

Functional Testing: Functions or features are tested against requirements mentioned in the Functional Requirement Specification (FRS) documents.

Acceptance Testing: Determines whether a product is suitable to be released for customer use. At this stage, developers will have to receive approval from testers and SMEs to push code changes from Staging to Production.

Integration Testing: Also called system testing, since testing occurs on the system level. These tests are performed once all software modules are integrated.

Advantages of Test Coverage

It reports on portions of the codebase that have not been covered by necessary test cases.

It helps to detect the areas of test cases that are useless for the current project. These cases are reported and can be eliminated to make the code lighter.

It helps developers create additional test cases as required. These additional test cases help ensure that test coverage is maximum.

It is useful for preventing defect leakage.

https://www.browserstack.com/guide/code-coverage-vs-test-coverage

Code Coverage Best Practices

Code coverage provides significant benefits to the developer workflow.

You must treat it with the understanding that it’s a lossy and indirect metric that compresses a lot of information into a single number so it should not be your only source of truth. Instead, use it in conjunction with other techniques to create a more holistic assessment of your testing efforts.

It is an open research question whether code coverage alone reduces defects,

but our experience shows that efforts in increasing code coverage can often lead to culture changes in engineering excellence that in the long run reduce defects

A high code coverage percentage does not guarantee high quality in the test coverage

Code coverage does not guarantee that the covered lines or branches have been tested correctly, it just guarantees that they have been executed by a test.

Be mindful of copy/pasting tests just for the sake of increasing coverage, or adding tests with little actual value, to comply with the number.

A better technique to assess whether you’re adequately exercising the lines your tests cover, and adequately asserting on failures, is mutation testing.

But a low code coverage number does guarantee that large areas of the product are going completely untested

There is no “ideal code coverage number” that universally applies to all products.

We cannot mandate every single team should have x% code coverage; this is a business decision best made by the owners of the product with domain-specific knowledge

In general code coverage of a lot of products is below the bar; we should aim at significantly improving code coverage across the board. Although there is no “ideal code coverage number,” at Google we offer the general guidelines of 60% as “acceptable”, 75% as “commendable” and 90% as “exemplary.”

We should not be obsessing on how to get from 90% code coverage to 95%.

More important than the percentage of lines covered is human judgment over the actual lines of code (and behaviors) that aren’t being covered

We have found out that embedding code coverage into your code review process makes code reviews faster and easier. Not all code is equally important, for example testing debug log lines is often not as important, so when developers can see not just the coverage number, but each covered line highlighted as part of the code review, they will make sure that the most important code is covered.

Just because your product has low code coverage doesn’t mean you can’t take concrete, incremental steps to improve it over time

But at the very least, you can adopt the ‘boy-scout rule’ (leave the campground cleaner than you found it). Over time, and incrementally, you will get to a healthy location.

Make sure that frequently changing code is covered.

per-commit coverage goals of 99% are reasonable, and 90% is a good lower threshold.

Unit test code coverage is only a piece of the puzzle. Integration/System test code coverage is important too. And the aggregate view of the coverage of all sources in your Pipeline (unit and integration) is paramount, as it gives you the bigger picture of how much of your code is not exercised by your test automation as it makes its way in your pipeline to a production environment.One thing you should be aware of is while unit tests have high correlation between executed and evaluated code, some of the coverage from integration tests and end-to-end tests is incidental and not deliberate. But incorporating code coverage from integration tests can help you avoid situations where you have a false sense of security that even though you’re not covering code in your unit tests, you think you’re covering it in your integration tests.

We should gate deployments that do not meet our code coverage standards

There are many mechanisms available: gate on coverage for all code vs gate on coverage to new code only; gate on a specific hard-coded code coverage number vs gate on delta from prior version, specific parts of the code to ignore or focus on. And then, commit to upholding these as a team. Drops in code coverage violating the gate should prevent the code from being checked in and reaching production.

https://testing.googleblog.com/2020/08/code-coverage-best-practices.html

Mutation testing

Mutation testing assesses test suite efficacy by inserting small faults into programs and measuring the ability of the test suite to detect them. It is widely considered the strongest test criterion in terms of finding the most faults and it subsumes a number of other coverage criteria.Furthermore, by reducing the number of mutants and carefully selecting only the most interesting ones we make it easier for humans to understand and evaluate the result of mutation analysis.

https://research.google/pubs/pub46584/

In computer programming and software testing, smoke testing (also confidence testing, sanity testing,[1] build verification test (BVT)[2][3][4] and build acceptance test) is preliminary testing to reveal simple failures severe enough to, for example, reject a prospective software release. Smoke tests are a subset of test cases that cover the most important functionality of a component or system, used to aid assessment of whether main functions of the software appear to work correctly.[1][2] When used to determine if a computer program should be subjected to further, more fine-grained testing, a smoke test may be called an intake test.[1] Alternatively, it is a set of tests run on each new build of a product to verify that the build is testable before the build is released into the hands of the test team

https://en.wikipedia.org/wiki/Smoke_testing_(software)

What are black, gray and white-box testing?

Pentesting assignments are classified based on the level of knowledge and access granted to the pentester at the beginning of the assignment.

Black-box testing

In a black-box testing assignment, the penetration tester is placed in the role of the average hacker, with no internal knowledge of the target system.

A black-box penetration test determines the vulnerabilities in a system that are exploitable from outside the network.

This means that black-box penetration testing relies on dynamic analysis of currently running programs and systems within the target network

The limited knowledge provided to the penetration tester makes black-box penetration tests the quickest to run, since the duration of the assignment largely depends on the tester’s ability to locate and exploit vulnerabilities in the target’s outward-facing services.

The major downside of this approach is that if the testers cannot breach the perimeter, any vulnerabilities of internal services remain undiscovered and unpatched.

Gray-box testing

a gray-box tester has the access and knowledge levels of a user, potentially with elevated privileges on a system.

Gray-box pentesters typically have some knowledge of a network’s internals, potentially including design and architecture documentation and an account internal to the network.

The purpose of gray-box pentesting is to provide a more focused and efficient assessment of a network’s security than a black-box assessment.

Using the design documentation for a network, pentesters can focus their assessment efforts on the systems with the greatest risk and value from the start, rather than spending time determining this information on their own.

An internal account on the system also allows testing of security inside the hardened perimeter and simulates an attacker with longer-term access to the network.

White-box testing

White-box testing goes by several different names, including clear-box, open-box, auxiliary and logic-driven testing.

penetration testers are given full access to source code, architecture documentation and so forth.

The main challenge with white-box testing is sifting through the massive amount of data available to identify potential points of weakness, making it the most time-consuming type of penetration testing.

Unlike black-box and gray-box testing, white-box penetration testers are able to perform static code analysis, making familiarity with source code analyzers, debuggers and similar tools important for this type of testing.

The close relationship between white-box pentesters and developers provides a high level of system knowledge but may affect tester’s behaviors, since they operate based on knowledge not available to hackers.

The main tradeoffs between black-box, gray-box and white-box penetration testing are the accuracy of the test and its speed, efficiency and coverage.

The purpose of penetration testing is to identify and patch the vulnerabilities that would be exploited by an attacker.

Therefore, the ideal form of penetration testing would be black-box, as the majority of attackers have no knowledge of the internal workings of their target network prior to launching their attack.

However, the average attacker has much more time to devote to their process than the average pentester, so the other types of penetration tests have been developed to decrease engagement time by increasing the level of information provided to the tester.

white-box testing,The concern with this type of pentesting engagement is that the increased information will cause testers to act in a way different from black-box hackers, potentially leading them to miss vulnerabilities that a less-informed attacker would exploit.

gray-box tests simulate the level of knowledge that a hacker with long-term access to a system would achieve through research and system footprinting.

Speed, efficiency and coverage

In general, black-box penetration testing is the fastest type of penetration test.

However, the limited information available to the testers increases the probability that vulnerabilities will be overlooked and decreases the efficiency of the test, since testers do not have the information necessary to target their attacks on the most high-value or likely vulnerable targets.

Gray-box testing,Access to design documentation allows testers to better focus their efforts and internal access to the network increases the coverage of the analysis.

White-box testing is the slowest and most comprehensive form of pentesting.The large amount of data available to pentesters requires time to process; however, the high level of access improves the probability that both internal and outward-facing vulnerabilities will be identified and remediated.

https://resources.infosecinstitute.com/topic/what-are-black-box-grey-box-and-white-box-penetration-testing/#gref

The pros and cons of gray box testing

Advantages of gray box testing:

Testing is still performed from the point of view of a user or attacker rather than a developer, which may help to uncover flaws that developers have missed.

Disadvantages of gray box testing:

Testers have no access to source code and may miss certain critical vulnerabilities.

Gray box testing may be redundant if the application developer has already run a similar test case.

https://www.veracode.com/security/gray-box-testing

1 comment:

Rahul singhNovember 11, 2021 at 11:23 AM
Your post is very great.I read this post. It’s very helpful. I will definitely go ahead and take advantage of this. You absolutely have wonderful stories. Cheers for sharing with us your blog. For more learning about data science visit at Data science course in Bangalore

Wednesday, June 27, 2012

Testing

1 comment: