Complexity and Big-O Notation

• Complexity and Big-O Notation

Be careful to differentiate between:

Performance: how much time/memory/disk/... is actually used when a program is run. This depends on the machine, compiler, etc. as well as the code.
Complexity: how do the resource requirements of a program or algorithm scale, i.e., what happens as the size of the problem being solved gets larger.

Complexity affects performance but not the other way around


Big-O Notation

We express complexity using big-O notation. For a problem of size N:

constant-time method is "order 1": O(1)
linear-time method is "order N": O(N)
quadratic-time method is "order N squared": O(N2)

How to Determine Complexities

In general, how can you determine the running time of a piece of code? The answer is that it depends on what kinds of statements are used.

1-Sequence of statements
statement 1;
statement 2;
  ...
statement k;

If each statement is "simple" (only involves basic operations) then the time for each statement is constant and the total time is also constant: O(1).

2-if-then-else statements

Here, either sequence 1 will execute, or sequence 2 will execute. Therefore, the worst-case time is the slowest of the two possibilities: max(time(sequence 1), time(sequence 2)). For example, if sequence 1 is O(N) and sequence 2 is O(1) the worst-case time for the whole if-then-else statement would be O(N)

3-for loops

for (i = 0; i < N; i++) {
    sequence of statements
}
The loop executes N times, so the sequence of statements also executes N times. Since we assume the statements are O(1), the total time for the for loop is N * O(1), which is O(N) overall.


4-Nested loops
First we'll consider loops where the number of iterations of the inner loop is independent of the value of the outer loop's index. For example:

for (i = 0; i < N; i++) {
    for (j = 0; j < M; j++) {
        sequence of statements
    }
}
The outer loop executes N times. Every time the outer loop executes, the inner loop executes M times. As a result, the statements in the inner loop execute a total of N * M times. Thus, the complexity is O(N * M). In a common special case where the stopping condition of the inner loop is j < N instead of j < M (i.e., the inner loop also executes N times), the total complexity for the two loops is O(N2).



Best-case and Average-case Complexity

Some methods may require different amounts of time on different calls, even when the problem size is the same for both calls. For example, consider the add method that adds an item to the end of the list. In the worst case (the array is full), that method requires time proportional to the number of items in the list (because it has to copy all of them into the new, larger array). However, when the array is not full, add will only have to copy one value into the array, so in that case its time is independent of the length of the list; i.e., constant time.


When do Constants Matter?

Recall that when we use big-O notation, we drop constants and low-order terms. This is because when the problem size gets sufficiently large, those terms don't matter. However, this means that two algorithms can have the same big-O time complexity, even though one is always faster than the other. For example, suppose algorithm 1 requires N2 time, and algorithm 2 requires 10 * N2 + N time. For both algorithms, the time is O(N2), but algorithm 1 will always be faster than algorithm 2. In this case, the constants and low-order terms do matter in terms of which algorithm is actually faster.



http://pages.cs.wisc.edu/~vernon/cs367/notes/3.COMPLEXITY.html

• Algorithmic Complexity

Algorithmic complexity is concerned about how fast or slow particular algorithm performs. We define complexity as a numerical function T(n) - time versus the input size n



Asymptotic Notations

The goal of computational complexity is to classify algorithms according to their performances. We will represent the time function T(n) using the "big-O" notation to express an algorithm runtime complexity.



Constant Time: O(1)

An algorithm is said to run in constant time if it requires the same amount of time regardless of the input size. Examples:

array: accessing any element
fixed-size stack: push and pop methods
fixed-size queue: enqueue and dequeue methods



Linear Time: O(n)

An algorithm is said to run in linear time if its time execution is directly proportional to the input size, i.e. time grows linearly as input size increases. Examples:

array: linear search, traversing, find minimum
ArrayList: contains method
queue: contains method


Logarithmic Time: O(log n)

An algorithm is said to run in logarithmic time if its time execution is proportional to the logarithm of the input size. Example:
binary search

Quadratic Time: O(n2)

An algorithm is said to run in logarithmic time if its time execution is proportional to the square of the input size. Examples:
bubble sort, selection sort, insertion sort

Definition of "big Omega"
We need the notation for the lower bound.


Definition of "big Theta"
To measure the complexity of a particular algorithm, means to find the upper and lower bounds

Analysis of Algorithms

The term analysis of algorithms is used to describe approaches to the study of the performance of algorithms. In this course we will perform the following types of analysis:
the worst-case runtime complexity of the algorithm is the function defined by the maximum number of steps taken on any instance of size a.
the best-case runtime complexity of the algorithm is the function defined by the minimum number of steps taken on any instance of size a.
the average case runtime complexity of the algorithm is the function defined by an average number of steps taken on any instance of size a.
the amortized runtime complexity of the algorithm is the function defined by a sequence of operations applied to the input of size a and averaged over time.


Example. Let us consider an algorithm of sequential searching in an array.of size n.
Its worst-case runtime complexity is O(n)
Its best-case runtime complexity is O(1)
Its average case runtime complexity is O(n/2)=O(n)


http://www.cs.cmu.edu/~adamchik/15-121/lectures/Algorithmic%20Complexity/complexity.html

• Sorting: Ep 07a - Big O Notation

http://www.youtube.com/watch?v=kRy8OlBAALk

• Big-O notation.mp4

http://www.youtube.com/watch?v=WoKlmaP3dto

• Algorithms: Big O, Growth Functions, and Worst Case Analysis (1/3)

http://www.youtube.com/watch?v=aPxauBqARGU&feature=relmfu

• Big-Oh, Omega and Theta notation

http://www.youtube.com/watch?v=pe250F4ttmE

• When solving a computer science problem there will usually be more than just one solution. These solutions will often be in the form of different algorithms, and you will generally want to compare the algorithms to see which one is more efficient. 

This is where Big O analysis helps – it gives us some basis for measuring the efficiency of an algorithm.
it measures the efficiency of an algorithm based on the time it takes for the algorithm to run as a function of the input size.

Basically, Big-O will want to express how many times the 'n' input items are 'touched'.
The word 'touched' can mean different things in different algorithms - in some algorithms it may mean the number of times a constant is multiplied by an input item, the number of times an input is added to a data structure, etc.

http://www.programmerinterview.com/index.php/data-structures/big-o-notation/

•  Big Omega 

 Big O describes the upper (worst case) bound, and Big Omega describes the lower (best case) bound.
 Insertion Sort has an upper bound of O(n^2) and a lower bound of Omega(n)
 http://www.linuxquestions.org/questions/programming-9/big-o-big-omega-and-big-theta-106102/

•  asymptotic bounds

 Big O, Big Omega, and Big Theta
 http://www.youtube.com/watch?v=6Ol2JbwoJp0
 http://xoax.net/comp_sci/crs/algorithms/lessons/Lesson6/

•  Algorithms Lesson 7: Analyzing Bubblesort 

 http://www.youtube.com/watch?v=ni_zk257Nqo


 http://www.cs.auckland.ac.nz/compsci220s1t/lectures/lecturenotes/GG-lectures/BigOhexamples.pdf

 http://www.programmerinterview.com/index.php/data-structures/big-o-notation/

 http://www.cforcoding.com/2009/07/plain-english-explanation-of-big-o.html

• This measure of efficiency is called asymptotic complexity,

and is used when dis-regarding certain terms of a function to express the efficiency of an algorithm
or when calculating a function is difficult or impossible and only approximations can be found.


The most commonly used notation for specifying asymptotic complexity—that is, for
estimating the rate of function growth—is the big-O notation

Given two positive-valued functions f and g,consider the following definition:
f(n) is O(g(n)) if there exist positive numbers cand Nsuch that f(n) =cg(n) for all n=N.

This definition reads:
f is big-O of g if there is a positive number c such that f is
not larger than cg for sufficiently large ns;
that is, for all ns larger than some number N.

The relationship between f and g can be expressed by stating either that g(n) is
an upper bound on the value of f(n) or that, in the long run,f grows at most as fast
as g.

Algorithms: Big O, Growth Functions, and Worst Case Analysis (1/3)

Big O Part 1 – Linear Complexity

describes the impact of increasing  the input data on the time taken for a program to run

big o space complexity

describes the impact of increasing the input data on the extra memory needed by the program

big o describes how the time is taken or how memory is used

big o describes complexity

common sense tells us that a program takes longer when there's more data to work on

linear search or sequential search

brute force approach

an item is searched in an unordered list

time is directly proportional to the amount of data  

linear search complexity

big o time complexity is linear, O(n)

Big O Part 2 – Constant Complexity

peek operations, peek over top item wo removing it

LIFO data structure

top is popped, the item is not removed, it is overwritten

no matter how much data is in the stack the time taken remains same

increasing the amount of data makes no difference to the time taken by push or pop operations

stack operations complexity

big o complexity is constant

O(1)

Big O Part 3 – Quadratic Complexity

the dominant term

n^3 is dominant to n^2

bubble sort

comparing,swapping

bubble sort complexity

for n data items, a simple implementation performs (n-1)*(n-1) operations

the dominant term is n^2

big o time complexity is quadratic, O(n^2)

enhanced bubble sort

for n data items, enhanced bubble sort algorithm performs (n-1)+(n-2)+(n-3).....+2+1

(n^2-n)/2

the dominant term is n^2, still O(n^2)

the big o complexity of enhanced bubble sort is still O(n^2)

alternative enhanced bubble sort

best vs worst case scenario

best case scenario, data is already sorted, O(n)

worst case scenario, data is in reverse order, O(n^2)

• DAA - Analysis of Algorithms

In theoretical analysis of algorithms, it is common to estimate their complexity in the asymptotic sense, i.e., to estimate the complexity function for arbitrarily large input. The term "analysis of algorithms" was coined by Donald Knuth.

Algorithm analysis is an important part of computational complexity theory, which provides theoretical estimation for the required resources of an algorithm to solve a specific computational problem. Most algorithms are designed to work with inputs of arbitrary length. Analysis of algorithms is the determination of the amount of time and space resources required to execute it.

Usually, the efficiency or running time of an algorithm is stated as a function relating the input length to the number of steps, known as time complexity, or volume of memory, known as space complexity.

The Need for Analysis

In this chapter, we will discuss the need for analysis of algorithms and how to choose a better algorithm for a particular problem as one computational problem can be solved by different algorithms.

By considering an algorithm for a specific problem, we can begin to develop pattern recognition so that similar types of problems can be solved by the help of this algorithm.

Algorithms are often quite different from one another, though the objective of these algorithms are the same. For example, we know that a set of numbers can be sorted using different algorithms. Number of comparisons performed by one algorithm may vary with others for the same input. Hence, time complexity of those algorithms may differ. At the same time, we need to calculate the memory space required by each algorithm.

Analysis of algorithm is the process of analyzing the problem-solving capability of the algorithm in terms of the time and size required (the size of memory for storage while implementation). However, the main concern of analysis of algorithms is the required time or performance. Generally, we perform the following types of analysis −

    Worst-case − The maximum number of steps taken on any instance of size a.

    Best-case − The minimum number of steps taken on any instance of size a.

    Average case − An average number of steps taken on any instance of size a.

    Amortized − A sequence of operations applied to the input of size a averaged over time.

In this context, if we compare bubble sort and merge sort. Bubble sort does not require additional memory, but merge sort requires additional space. Though time complexity of bubble sort is higher compared to merge sort, we may need to apply bubble sort if the program needs to run in an environment, where memory is very limited.

https://www.tutorialspoint.com/design_and_analysis_of_algorithms/analysis_of_algorithms.htm

• Asymptotic Notations and Apriori Analysis

Complexity of an algorithm is analyzed in two perspectives: Time and Space.

Time Complexity

It’s a function describing the amount of time required to run an algorithm in terms of the size of the input. "Time" can mean the number of memory accesses performed, the number of comparisons between integers, the number of times some inner loop is executed, or some other natural unit related to the amount of real time the algorithm will take.

Space Complexity

It’s a function describing the amount of memory an algorithm takes in terms of the size of input to the algorithm. We often speak of "extra" memory needed, not counting the memory needed to store the input itself. Again, we use natural (but fixed-length) units to measure this.

Space complexity is sometimes ignored because the space used is minimal and/or obvious, however sometimes it becomes as important an issue as time.

Asymptotic Notations

Execution time of an algorithm depends on the instruction set, processor speed, disk I/O speed, etc. Hence, we estimate the efficiency of an algorithm asymptotically.

Different types of asymptotic notations are used to represent the complexity of an algorithm. Following asymptotic notations are used to calculate the running time complexity of an algorithm.

    O − Big Oh

    Ω − Big omega

    θ − Big theta

    o − Little Oh

    ω − Little omega

https://www.tutorialspoint.com/design_and_analysis_of_algorithms/design_and_analysis_of_algorithms_asymptotic_notations_apriori.htm