service discovery / clusterization

• Decommissioning Otto

Otto was created to provide a high-level abstraction to both develop and deploy applications. Its goal was to unify HashiCorp tools across the application delivery spectrum, including Vagrant to manage development environments, Packer to build images, Terraform to provision images, Consul to connect and monitor applications, Nomad to deploy applications, and Vault to secure applications and infrastructure

https://www.ottoproject.io/

•  how Microservices work with Consul

 how the following components affect Consul.
    Using docker
    Building Registrator for Service Discovery
    Using rkt and Nomad

the consul server will wait until there are 3 peers connected before self-bootstrapping and becoming a working cluster

Building Registrator for Service Discovery
Registrator automatically registers and deregisters services for any Docker container by inspecting containers as they come online.
The Registrator we are about to use currently supports pluggable service registries, which currently includes Consul, Etcd and SkyDNS2

Using rkt and Nomad
The rkt is another container-based service
The main reason for building rkt was to improve the security that was one of the crisis issues for Docker back when it was still in development in 2013-14.
As for Consul, we can use the Rkt Registrator for working on service discovery with Consul.

Nomad is a tool for managing a cluster of machines and running applications on them.
It is similar to Mesos or Kubernetes.
By default, Nomad covers the Docker and rkt driver within itself.
https://www.tutorialspoint.com/consul/consul_working_with_microservices.htm

• Within each datacenter, we have a mixture of clients and servers. It is expected that there be between three to five servers. This strikes a balance between availability in the case of failure and performance, as consensus gets progressively slower as more machines are added. However, there is no limit to the number of clients, and they can easily scale into the thousands or tens of thousands.
  
  All the nodes that are in a datacenter participate in a gossip protocol.
  This means there is a gossip pool that contains all the nodes for a given datacenter.
  first, there is no need to configure clients with the addresses of servers; discovery is done automatically
  Second, the work of detecting node failures is not placed on the servers but is distributed. This makes failure detection much more scalable than naive heartbeating schemes.
  Thirdly, it is used as a messaging layer to notify when important events such as leader election take place.
  
  The servers in each datacenter are all part of a single Raft peer set. This means that they work together to elect a single leader, a selected server which has extra duties. The leader is responsible for processing all queries and transactions. Transactions must also be replicated to all peers as part of the consensus protocol. Because of this requirement, when a non-leader server receives an RPC request, it forwards it to the cluster leader.
  
  The server nodes also operate as part of a WAN gossip pool. This pool is different from the LAN pool as it is optimized for the higher latency of the internet and is expected to contain only other Consul server nodes.
  The purpose of this pool is to allow datacenters to discover each other in a low-touch manner.
  Bringing a new datacenter online is as easy as joining the existing WAN gossip pool.
  Because the servers are all operating in this pool, it also enables cross-datacenter requests.
  
  In general, data is not replicated between different Consul datacenters.
  When a request is made for a resource in another datacenter, the local Consul servers forward an RPC request to the remote Consul servers for that resource and return the results
  There are some special situations where a limited subset of data can be replicated, such as with Consul's built-in ACL replication capability, or external tools like consul-replicate.
  
  https://www.consul.io/docs/internals/architecture.html
  
  
  
  • Basic Federation with the WAN Gossip Pool 
  
  One of the key features of Consul is its support for multiple datacenters.
  The architecture of Consul is designed to promote a low coupling of datacenters so that connectivity issues or failure of any datacenter does not impact the availability of Consul in other datacenters.
  This means each datacenter runs independently, each having a dedicated group of servers and a private LAN gossip pool.
  https://www.consul.io/docs/guides/datacenters.html
  
  
  
  • Quorum - A quorum is a majority of members from a peer set: for a set of size n, quorum requires at least (n/2)+1 members. For example, if there are 5 members in the peer set, we would need 3 nodes to form a quorum. If a quorum of nodes is unavailable for any reason, the cluster becomes unavailable and no new logs can be committed.
  
  https://www.consul.io/docs/internals/consensus.html
  
  
  
  
  • Consul uses a consensus protocol to provide Consistency (as defined by CAP). The consensus protocol is based on "Raft: 
  
  https://www.consul.io/docs/internals/consensus.html
  
  
  
  • Raft is a consensus algorithm that is based on Paxos. Compared to Paxos, Raft is designed to have fewer states and a simpler, more understandable algorithm.
  
  https://www.consul.io/docs/internals/consensus.html

• Gossip Protocol

Serf uses a gossip protocol to broadcast messages to the cluster.
https://www.serf.io/docs/internals/gossip.html

• Serf is a decentralized solution for cluster membership, failure detection, and orchestration. Serf is in use in some huge deployments (more than 10,000 machines in a single cluster), and powers Consul and Nomad.

This release brings improvements in Serf's gossip protocol which provide better robustness for applications that rely on Serf to detect the health of nodes in a cluster
https://www.hashicorp.com/blog/serf-0-8

• Thanos - a Scalable Prometheus with Unlimited Storage

Thanos' architecture introduces a central query layer across all the servers via a sidecar component which sits alongside each Prometheus server, and a central Querier component that responds to PromQL queries. This makes up a Thanos deployment. Inter-component communication is via the memberlist gossip protocol.
https://www.infoq.com/news/2018/06/thanos-scalable-prometheus

• Highly available Prometheus setup with long term storage capabilities. 

Thanos is a set of components that can be composed into a highly available metric system with unlimited storage capacity. It can be added seamlessly on top of existing Prometheus deployments and leverages the Prometheus 2.0 storage format to cost-efficiently store historical metric data in any object storage while retaining fast query latencies. Additionally, it provides a global query view across all Prometheus installations and can merge data from Prometheus HA pairs on the fly.
https://github.com/improbable-eng/thanos

• Consul is an important service discovery tool in the world of Devops. 

https://www.tutorialspoint.com/consul/index.htm

Consul is a Hashicorp based tool for discovering and configuring a variety of different services in your infrastructure
It is based and built on Golang.
One of the core reasons to build Consul was to maintain the services present in the distributed systems
https://www.tutorialspoint.com/consul/index.htm


Service Discovery − Using either DNS or HTTP, applications can easily find the services they depend upon.
Health Check Status It is used by the service discovery components to route traffic away from unhealthy hosts.
Key/Value Store −dynamic configuration, feature flagging, coordination, leader election, etc
Multi Datacenter Deployment − Consul supports multiple datacenters. It is used for building additional layers of abstraction to grow to multiple regions.


Service Discovery
It is defined as the detection of different services and network protocols using which a service is found.
The usage of service discovery comes in as a boon for distributed systems.
This is one of the main problems, which are faced by today's large-scale industries with the advancement of distributed systems in their environment.

Comparison with Etcd and Zookeeper


https://www.tutorialspoint.com/consul/consul_introduction.htm

 there are three different servers, which are managed by Consul. The working architecture works by the using raft algorithm, which helps us in electing a leader out of the three different servers.
 Each server interacts with its own client using the concept of RPC. The Communication between the Clients is possible due to Gossip Protocol as mentioned below
 The Communication with the internet facility can be made available using TCP or gossip method of communication

 Raft Algorithm
Raft is a consensus algorithm for managing a replicated log. It relies on the principle of CAP Theorem, which states that in the presence of a network partition, one has to choose between consistency and availability.

A Raft Cluster contains several servers, usually in the odd number count.
For example, if we have five servers, it will allow the system to tolerate two failures.
At any given time, each server is in one of the three states: Leader, Follower, or Candidate.
In a normal operation, there is exactly one leader and all of the other servers are followers. These followers are in a passive state, i.e. they issue no requests on their own, but simply respond to requests from leaders and the candidate.

There are two types of protocol in Consul, which are called as −
    Consensus Protocol and
    Gossip Protocol

Consensus Protocol
Consensus protocol is used by Consul to provide Consistency as described by the CAP Theorem.

Gossip Protocol
The gossip protocol can be used to manage membership, send and receive messages across the cluster.
In consul, the usage of gossip protocol occurs in two ways, WAN (Wireless Area Network) and LAN (Local Area Network).

There are three known libraries, which can implement a Gossip Algorithm to discover nodes in a peer-to-peer network −
    teknek-gossip − It works with UDP and is written in Java.
    gossip-python − It utilizes the TCP stack and it is possible to share data via the constructed network as well.
    Smudge − It is written in Go and uses UDP to exchange status information.

Gossip protocols have also been used for achieving and maintaining a distributed database consistency or with other types of data in consistent states, counting the number of nodes in a network of unknown size, spreading news robustly, organizing nodes, etc

Remote Procedure Calls
a protocol that one program uses to request a service from another program
Before RPC, Consul used to have only TCP and UDP based connections, which were good with most systems, but not in the case of distributed systems. RPC solves such problems by reducing the time-period of transfer of packet information from one place to another.
In this area, GRPC by Google is a great tool to look forward in case one wishes to observe benchmarks and compare performance.
 https://www.tutorialspoint.com/consul/consul_architecture.htm

•  A high performance, open-source universal RPC framework

 Install runtime and dev environments with a single line and also scale to millions of RPCs per second with the framework
 https://grpc.io/

• ZooKeeper

A high-performance coordination service for distributed applications.ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications. Each time they are implemented there is a lot of work that goes into fixing the bugs and race conditions that are inevitable. Because of the difficulty of implementing these kinds of services, applications initially usually skimp on them , which make them brittle in the presence of change and difficult to manage. Even when done correctly, different implementations of these services lead to management complexity when the applications are deployed
http://zookeeper.apache.org/

• etcd is a distributed key-value store that provides a reliable way to store data across a cluster of machines. It’s open-source and available on GitHub. etcd gracefully handles leader elections during network partitions and will tolerate machine failure, including the leader.

https://coreos.com/etcd/

• Setup Etcd Cluster on CentOS 7/8 / Ubuntu 18.04/16.04 / Debian 10/9

etcd is a distributed and reliable key-value store for the most critical data of a distributed system. It is written in Go and uses the Raft consensus algorithm to manage a highly-available replicated log.

Etcd is designed to be:
Simple: well-defined, user-facing API (gRPC)
Secure: automatic TLS with optional client cert authentication
Fast: benchmarked 10,000 writes/sec
Reliable: properly distributed using Raft
https://computingforgeeks.com/setup-etcd-cluster-on-centos-debian-ubuntu/



• Gossip protocol

a procedure or process of computer–computer communication that is based on the way social networks disseminate information or how epidemics spread.
It is a communication protocol.
Modern distributed systems often use gossip protocols to solve problems that might be difficult to solve in other ways, either because the underlying network has an inconvenient structure, is extremely large, or because gossip solutions are the most efficient ones available.
https://en.wikipedia.org/wiki/Gossip_protocol

• Gossip Protocols are executed periodically, that is, in cycles. That's what the word cycle in the simulator means. It will rotate in every cycle.

https://flopezluis.github.io/gossip-simulator/

• In large distributed systems, knowing the state of the whole system is a difficult task that becomes harder as you increment the number of nodes.

Since the underlying network is a problem, you can’t rely on hardware solutions, such as multicast, as they wouldn’t be available in the cloud.
https://conferences.oreilly.com/velocity/vl-eu/public/schedule/detail/71020

• The Gossip Protocol is a method to resolve this communication chaos. In Cassandra, when one node talks to another, the node which is expected to respond, not only provides information about its status,but also provides information about the nodes that it had communicated with before. Through this process, there is a reduction in network log, more information is kept and efficiency of information gathering increases. The main feature of the protocol is to provide the latest information of any node respectively.

An important feature of Gossip Protocol is Failure Detection. Basically, when two nodes communicate with one another; for instance, Node A to Node B, then Node A sends a message ‘gossipdigestsynmessage’, which is very similar to TCP protocol to Node B. Here, Node B, once receives the message, sends an acknowledgement message ‘ack’, and then Node A responds with an acknowledgement message to Node B’s ‘ack’ message. This is known as the 3 way handshake.
https://www.edureka.co/blog/gossip-protocol-in-cassandra/