Code Coverage

• In computer science, test coverage is a measure used to describe the degree to which the source code of a program is executed when a particular test suite runs

A program with high test coverage, measured as a percentage, has had more of its source code executed during testing, which suggests it has a lower chance of containing undetected software bugs compared to a program with low test coverage
Basic coverage criteria
There are a number of coverage criteria, the main ones being:
    Function coverage – Has each function (or subroutine) in the program been called?
    Statement coverage – Has each statement in the program been executed?
    Edge coverage – has every edge in the Control flow graph been executed?
    Branch coverage – Has each branch (also called DD-path) of each control structure (such as in if and case statements) been executed? For example, given an if statement, have both the true and false branches been executed? Notice that this one is a subset of Edge coverage.
    Condition coverage (or predicate coverage) – Has each Boolean sub-expression evaluated both to true and false?
https://en.wikipedia.org/wiki/Code_coverage

• What is Code Coverage?

Code coverage is the percentage of code which is covered by automated tests. Code coverage measurement simply determines which statements in a body of code have been executed through a test run, and which statements have not. In general, a code coverage system collects information about the running program and then combines that with source information to generate a report on the test suite's code coverage.
Code coverage is part of a feedback loop in the development process. As tests are developed, code coverage highlights aspects of the code which may not be adequately tested and which require additional testing. This loop will continue until coverage meets some specified target.

Why Measure Code Coverage?
It is well understood that unit testing improves the quality and predictability of your software releases
Measuring code coverage can keep your testing up to the standards you require.

Do you know, however, how well your unit tests actually test your code?
How many tests are enough?
Do you need more tests?

In summary, we measure code coverage for the following reasons:
    To know how well our tests actually test our code
    To know whether we have enough testing in place
    To maintain the test quality over the lifecycle of a project


https://confluence.atlassian.com/clover/about-code-coverage-71599496.html

• Use code coverage to determine how much code is being tested

To determine what proportion of your project's code is actually being tested by coded tests such as unit tests, you can use the code coverage feature of Visual Studio. To guard effectively against bugs, your tests should exercise or 'cover' a large proportion of your code
https://docs.microsoft.com/en-us/visualstudio/test/using-code-coverage-to-determine-how-much-code-is-being-tested?view=vs-2017

• Code Coverage vs. Test Coverage: Pros and Cons

Code Coverage
This metric aims to measure number of lines covered by the test cases. It reports total number of lines in the code and number of lines executed by tests. Think of it as the degree to which the source code of a program is executed when a test suite runs. The intent is, the higher the code coverage, the lower the chance of having undetected software bugs.
Subtypes

Coverage is further split into many subtypes – function coverage, branch coverage, condition coverage, loop coverage, statement coverage and parameter value coverage.

Let me define the subtypes for the sake of clarity.

    If tests cover all the function calls in code, function coverage is said to be 100%.
    Similarly, if all branches in the code, i.e. all the if – else conditions have been tested with every possible input, then branch coverage is said to be 100%.
    If all loop statements in the code are executed then loop coverage and statement coverage would be 100% respectively

For Python, a popular tool for code coverage measurement is Coverage.py
A measurement tool widely used for Go is gocov.


Test Coverage

Test Coverage aims to measure of the effectiveness of testing in a qualitative manner. It determines whether the test cases are covering entire functional requirements. You can think of it as a kind of black box testing, where test cases are not written based on code but based on user requirements or expected functionality.
Subtypes
The common mechanisms used for test coverage measurement are unit testing, functional testing, performance testing, integration or system testing and acceptance testing.

Unit tests are written at a granular level to check if a function/method performs as expected.
In functional testing, each functionality mentioned in the requirement document is tested.
Performance testing is generally a way to stress test the code for its responsiveness and stability at different workloads.
Integration testing or system testing is done to test if completely integrated product works in expected manner.
Acceptance testing is generally done at the end of the development cycle. For acceptance testing, the product is handed over to the stakeholders and tested by them to determine whether the product is in acceptable state.

For Python, framework called PyUnit is available.
JUnit, the well known Java testing framework serves as a foundation to launch tests on the Java Virtual Machine.
Go also offers a built in testing package for programs letting you easily write, organize and run tests


Code Coverage vs Test Coverage
So, now we know that code coverage is a measure of how much code is executed during testing, while test coverage is a measure of how much of the feature set is covered with tests.

Code Coverage Pros and Cons
Pros:
Helps capture bugs in program flow.
Cons:
Most Code coverage tools are relevant for Unit test only. It is essential to track all test types coverage (unit, API, System and even manual).

Test Coverage Pros and Cons
As a black-box testing methodology this approach is aligned to the requirement specifications and has minimal interaction with code itself.
Cons:
Most of the tasks in such testing methodology are manual.

Function and feature are different. Covering a single feature will cover several functions, while the required effort may be similar.

https://www.sealights.io/test-metrics/code-coverage-vs-test-coverage-pros-and-cons

• Code coverage is a metric that can help you understand how much of your source is tested

Code coverage tools will use one or more criteria to determine how your code was exercised or not during the execution of your test suite.

Function coverage: how many of the functions defined have been called.
Statement coverage: how many of the statements in the program have been executed.
Branches coverage: how many of the branches of the control structures (if statements for instance) have been executed.
Condition coverage: how many of the boolean sub-expressions have been tested for a true and a false value.
Line coverage: how many of lines of source code have been tested

Code coverage tools can help you understand where you should focus your attention next, but they won't tell you if your existing tests are robust enough for unexpected behaviors.
Achieving great coverage is an excellent goal, but it should be paired with having a robust test suite that can ensure that individual classes are not broken as well as verify the integrity of the system
When you've established your continuous integration (CI) workflow you can start failing the tests if you don't reach a high enough percentage of coverage. Of course, as we said it earlier, it would be unreasonable to set the failure threshold too high, and 90% coverage is likely to cause your build to fail a lot. If your goal is 80% coverage, you might consider setting a failure threshold at 70% as a safety net for your CI culture.
Once again, be careful to avoid sending the wrong message as pressuring your team to reach good coverage might lead to bad testing practices
https://www.atlassian.com/continuous-delivery/software-testing/code-coverage

• Linting is the process of checking the source code for Programmatic as well as Stylistic errors. This is most helpful in identifying some common and uncommon mistakes that are made during coding
A Lint or a Linter is a program that supports linting (verifying code quality). They are available for most languages like JavaScript, CSS, HTML, Python, etc..
https://stackoverflow.com/questions/8503559/what-is-linting/8503586


• A linter or lint refers to tools that analyze source code to flag programming errors, bugs, stylistic errors, and suspicious constructs.The term is originated from a Unix utility that examined C language source code.

https://en.wikipedia.org/wiki/Lint_(software)


• EclEmma

EclEmma is an Eclipse plug-in that generates code coverage reports and provide simple trace information about test cases
Coverage is defined as a measure of the completeness of the set of test cases.
This definition means that the more source code that is executed by your test cases the better.
There are four types of coverage:
    Method coverage: how many of your methods have been called by your test cases?
    Statement coverage: how many of your statements have been run by your test cases?
    Decision/Branch coverage: how many decision points (if statements) have evaluated both true and false?
    Condition coverage: how many Boolean sub-expressions have been evaluated both true and false at a decision point?
EclEmma may be used on top of JUnit.
EclEmma runs the JUnit test cases and generates the coverage report from the execution.
EclEmma provides overall, package, and file level information on the statement.
http://agile.csc.ncsu.edu/SEMaterials/tutorials/eclemma/

• Coverage.py

Coverage measurement is typically used to gauge the effectiveness of tests. It can show which parts of your code are being exercised by tests, and which are not.
https://coverage.readthedocs.io/en/v4.5.x/

• gocov

Coverage reporting tool for The Go Programming Language
https://github.com/axw/gocov

• EMMA

EMMA is a fast Java code coverage tool based on bytecode instrumentation. It differs from the existing tools by enabling coverage profiling on large scale enterprise software projects with simultaneous emphasis on fast individual development.
http://sourceforge.net/projects/emma/

• Crap4j 

Crap4j report screenshot. Crap4j is a Java implementation of the CRAP (Change Risk Analysis and Predictions) software metric. The CRAP metric combines cyclomatic complexity and code coverage from automated tests (e.g. JUnit tests) to help you identify code that might be particularly difficult to understand, test, or maintain
www.crap4j.org

• Cobertura

Cobertura is a Java code coverage reporting tool. It is based on jcoverage and works on any platform with Java 5 or higher. You can use Cobertura with ant, maven or command line access.
Cobertura is a free Java tool that calculates the percentage of code accessed by tests. It can be used to identify which parts of your Java program are lacking test coverage. It is based on jcoverage.
http://cobertura.github.io/cobertura/

• djUnit

djUnit uses jcoverage to generate a coverage report of the performed tests, and it can be checked on Eclipse.
The results are shown on djUnit coverage view , task list and java source editor.

http://works.dgic.co.jp/djunit/

• Lint4j

Lint4j ("Lint for Java") is a static Java source and byte code analyzer that detects locking and threading issues, performance and scalability problems, and checks complex contracts such as Java serialization by performing type, data flow, and lock graph analysis.

http://www.jutils.com/

• JaCoCo - Java Code Coverage Library

https://www.jacoco.org/jacoco/trunk/index.html