data lake

• ChatGPT

A data lake is a centralized repository designed to store large volumes of raw, unstructured, and structured data in its native format. It is a critical component in modern data architectures, particularly for handling big data and enabling advanced analytics. Here’s a comprehensive explanation:

Key Concepts of a Data Lake

1. Storage of Raw Data

• Raw Data: Data lakes store data in its raw form, without the need for pre-processing or transformation before storage. This can include structured data (like relational databases), semi-structured data (like JSON or XML), and unstructured data (like text documents, images, or videos).

• Flexibility: Unlike traditional databases or data warehouses, data lakes do not enforce a schema-on-write, meaning that data is stored as-is and schema is applied when the data is read (schema-on-read).

• 2. Scalability

  • Big Data Handling: Data lakes are designed to handle large volumes of data, scaling horizontally by adding more servers or storage as needed. This makes them suitable for big data applications and scenarios where data volume grows rapidly.

  3. Storage Cost Efficiency

  • Cost-Effective Storage: Storing large amounts of data in a data lake is often more cost-effective compared to traditional databases or data warehouses, especially when using cloud-based solutions.

  4. Data Accessibility

  • Centralized Repository: Data lakes act as a central repository where all data, regardless of type or source, can be stored and accessed. This provides a single source of truth for data analysis and business intelligence.

  5. Advanced Analytics and Machine Learning

  • Data Exploration: Analysts and data scientists can explore and analyze data from a data lake using various tools, including SQL queries, machine learning algorithms, and big data processing frameworks.

  • Integration with Analytical Tools: Data lakes integrate with data processing and analytics tools like Apache Hadoop, Apache Spark, and other big data frameworks, enabling complex data processing and advanced analytics.

 

Components of a Data Lake

1. Data Ingestion:

   • Mechanisms for collecting and importing data from various sources into the data lake. This can include batch processing, real-time streaming, and data integration tools.

2. Data Storage:

   • Storage infrastructure that can handle different types of data formats and volumes. In cloud environments, services like Amazon S3, Azure Data Lake Storage, or Google Cloud Storage are commonly used.

3. Data Management:

   • Tools and frameworks for organizing, cataloging, and managing the data stored in the lake. This includes metadata management, data governance, and data quality tools.

4. Data Processing:

   • Technologies and frameworks for processing and analyzing the data. This can include data transformation, ETL (Extract, Transform, Load) processes, and big data processing engines.

5. Data Access and Security:

• Mechanisms to ensure secure access to the data, including authentication, authorization, and encryption. Data lakes implement policies and controls to protect sensitive information and manage user access.

Advantages of a Data Lake

• Flexibility: Can store diverse data types and formats, allowing organizations to adapt to changing data needs and use cases.
• Scalability: Easily scalable to accommodate growing data volumes and processing requirements.
• Cost Efficiency: Typically more cost-effective for large-scale data storage compared to traditional databases.
• Advanced Analytics: Supports advanced analytics and machine learning by providing access to large volumes of raw data.

Challenges of a Data Lake

• Data Quality and Governance: Managing and ensuring the quality of data can be challenging due to the volume and diversity of data stored.
• Complexity: Requires robust data management and processing frameworks to effectively handle and analyze the data.
• Security: Ensuring data security and compliance can be complex given the wide range of data types and sources.

Summary

A data lake is a powerful tool for organizations to store and manage large volumes of raw data from various sources. It provides the flexibility to handle diverse data types, supports scalability and advanced analytics, and offers cost-effective storage solutions. However, managing data quality, governance, and security remains essential to leveraging the full potential of a data lake.

• What is a data lake?

A data lake is a centralized repository designed to store, process, and secure large amounts of structured, semistructured, and unstructured data. It can store data in its native format and process any variety of it, ignoring size limits.

A data lake provides a scalable and secure platform that allows enterprises to: ingest any data from any system at any speed—even if the data comes from on-premises, cloud, or edge-computing systems; store any type or volume of data in full fidelity; process data in real time or batch mode; and analyze data using SQL, Python, R, or any other language, third-party data, or analytics application.

https://cloud.google.com/learn/what-is-a-data-lake#:~:text=A%20data%20lake%20is%20a,of%20it%2C%20ignoring%20size%20limits.

• Four key differences between a data lake and a data warehouse

Data Lake Data Warehouse

Data Structure Raw Processed

Purpose of Data Not yet determined Currently in use

Users Data scientists Business professionals

Accessibility Highly accessible and quick to update More complicated and costly to make changes

Organizations often need both. Data lakes were born out of the need to harness big data and benefit from the raw, granular structured and unstructured data for machine learning, but there is still a need to create data warehouses for analytics use by business users

https://www.talend.com/resources/data-lake-vs-data-warehouse/#:~:text=Data%20lakes%20and%20data%20warehouses,processed%20for%20a%20specific%20purpose.

Time Series Database

• Data is stored exactly as you give it.Runs on Hadoop and HBase.Generate graphs from the GUI

http://opentsdb.net/

• OpenTSDB is a distributed, scalable Time Series Database (TSDB) written on

top of HBase.
https://github.com/OpenTSDB/opentsdb

kubernetes

• A recent performance benchmark completed by Intel and BlueData using the BigBench benchmarking kit has shown that the performance ratios for container-based Hadoop workloads on BlueData EPIC are equal to and in some cases, better than bare-metal Hadoop

Under the hood, the BlueData used several enhancements to boosts the I/O performance and scalability of container-based clusters.

container-based Spark cluster vs. bare-metal Spark cluster.
For instance, scatter/gather pattern can be used to implement a MapReduce like batch processing architecture on top of Kubernetes. Similarly, event-driven stream data processing is a lot easier to implement as microservices running on top Kubernetes.

a distributed Pachyderm File System (PFS) and a data-aware scheduler Pachyderm Pipeline System (PPS) on top of Kubernetes.
Pachyderm uses default Kubernetes scheduler to implement fault-tolerance and incremental processing.
In addition, for FPS Pachyderm utilizes a copy-on-write paradigm which inspired by Git.
Pachyderm is applying version control to your data as it's processed which processing jobs run on only the diff.

Custom schedulers

Kubernetes custom schedulers specifically optimised for big data workloads
Kubernetes scheduler is responsible for scheduling pods onto nodes. Kubernetes ships with a default scheduler which provides a range of scheduling features. To schedule pods onto nodes, Kubernetes default scheduler considers several factors including individual and collective resource requirements, quality of service requirements, hardware constraints, affinity or anti-affinity specifications, data locality, inter-workload interference and so on. Using default scheduler's node affinity feature you can ensure that certain pods only schedule on nodes with specialized hardware like GPU, memory-optimised, I/O optimised etc. Similarly, pods affinity features allow you to place pods relative to one another.
Kubernetes allows you to run multiple schedulers simultaneously.

the performance of Spark with native Kubernetes scheduler can be improved by running HDFS inside Kubernetes
This enables HDFS data locality by discovering the mapping of Kubernetes containers to physical nodes to HDFS datanode daemons
Basically having HDFS in Kubernetes makes schedule data-aware.
It is possible to use YARN as Kubernetes custom scheduler.
Heron is a real-time, distributed stream processing engine developed at Twitter.
It can be considered as a drop-in replacement for Apache Storm.
Just like Apache Storm, Heron has a concept of topology.
A topology is a directed acyclic graph (DAG) used to process streams of data and it can be stateless or stateful.
Heron topology is essentially a set of pods that can be scheduled by Kubernetes.
Heron scheduler converts packing plan for a topology into pod definitions which is then submitted to Kubernetes scheduler via APIs
A topology can be updated (scale up or down based on load) without having to build a new JAR to submit to the cluster.

Storage provisioning
Storage options have been another big roadblock in porting data workloads on Kubernetes particularly for stateful data workloads like Zookeeper, Cassandra, etc.
These new Kubernetes storage options have enabled us to deploy more fault-tolerance stateful data workloads on Kubernetes without the risk of data loss. For instance, by levering PersistentVolumes, a custom Cassandra Seed Provider, and StatefulSets we can provide a resilient installation of Cassandra
https://www.abhishek-tiwari.com/kubernetes-for-big-data-workloads/

• What is a container?

It’s similar to a virtual machine (VM), but it avoids a great deal of the trouble because it virtualizes the operating system (OS) rather than the underlying hardware.

This enables engineers to quickly develop applications that will run consistently across a large number of machines and software environments.

What is Docker?

The Docker Container Platform is an excellent tool for building and deploying containerized applications. 

 The platform helps developers easily isolate software into containers as they create it. It’s also an effective way to prepare existing applications for the cloud.

 What is Kubernetes?

 Kubernetes is an open source solution for container orchestration.

 Kubernetes and Docker: Finding your best container solution

 While Docker does have its own container orchestration solution called Docker Swarm, Kubernetes and Docker mostly solve different problems and thus can coexist. Later versions of Docker even have built-in integration with Kubernetes.

https://www.ibm.com/blogs/cloud-computing/2018/07/30/kubernetes-docker-vs/

Tiered Storage

• tiered storage

Tiered storage is a way to assign different categories of data to various types of storage media with the objective of reducing the total cost of storage.
A tiered storage architecture places data in a hierarchy according to its business value.
Tiers are determined by performance and cost of the media, and data is ranked by how often users access it.

Tier 1 storage
Tier 1 data includes mission-critical applications, recently accessed data or top-secret files. This data might be stored on expensive, high-quality media, such as double-parity RAID.

Tier 2 and Tier 3 storage
Even as primary operations are being completed, Tier 1 data is usually written simultaneously to a secondary tier of disk-based backup appliances or to magnetic tape. Data centers deploy a backup tier to aid business continuity and disaster recovery (BC/DR) through fast restores of key files and storage hardware.
Data on Tier 2 storage usually contains historical financial information, cold data and classified files. This data is preserved on lower-cost media in a conventional storage area network (SAN)

Tiering vs. caching
Storage tiering and caching technologies are often used interchangeably -- especially when dealing with flash media -- but they are different processes. Tiered data resides on one media type at any time but moves between media as data access patterns change.

https://searchstorage.techtarget.com/definition/tiered-storage

• Flexible Storage Throughout the Data Lifecycle

add-on feature lets you store and manage data in different tiers based on cost and performance trade-offs—whether it’s flash storage, traditional local or shared disk storage, HDFS, or Amazon cloud storage. With Tiered Storage, your data is easily migrated between tiers without any ETL, additional software, or expensive infrastructure changes. This allows you to easily balance performance and capacity through the lifecycle of your data—meeting performance SLAs, making data governance easier, and meeting compliance requirements.

https://www.marklogic.com/product/marklogic-database-overview/database-features/tiered-storage/

Distributed SQL Database

• Distributed SQL

Scale your SQL Database across 100s of nodes.Apache Ignite comes with ANSI-99 compliant, horizontally scalable and fault-tolerant distributed SQL database.
As an​ SQL database, Ignite supports all DML commands including SELECT, UPDATE, INSERT, and DELETE queries. It also implements a subset of DDL commands relevant for distributed systems.
https://apacheignite.readme.io/docs/distributed-sql

• TRADITIONAL RELATIONAL SQL DATABASES

like Oracle and Microsoft SQL Server are great for on-premises, single data center environments, but fail to adapt to the more distributed, less predictable, and higher in volume workloads found in today’s enterprise.
NOSQL SOLUTIONS
like MongoDB, Couchbase, and DataStax solve for on-demand scale and continuous availability needs, but force organizations to rewrite SQL-based applications, resulting in the loss of the data consistency that transactional systems require
NEWSQLPRODUCTS
such as VoltDB, Clustrix, and MemSQL maintain SQL as they scale, but are often limited to highly specialized use cases such as streaming or batch analytic applications

In short, if you are looking to retain your traditional SQL database benefits and gain new cloud efficiencies, then a distributed SQL database is the right option to consider.
https://www.nuodb.com/digging-distributed-sql

• Apache Ignite™ is a memory-centric distributed database, caching, and processing platform for

transactional, analytical, and streaming workloads delivering in-memory speeds at petabyte scale
https://ignite.apache.org/

• F1: A Distributed SQL Database That Scales

F1 is a hybrid database that combines high availability, the scalability of NoSQL systems like Bigtable, and the consistency and usability of traditional SQL databases.
https://ai.google/research/pubs/pub41344

• Actordb

Distributed SQL database with linear scalability
http://www.actordb.com/

• NewSQL is a class of modern relational database management systems that seek to provide the same scalable performance of NoSQL systems for online transaction processing (OLTP) read-write workloads while still maintaining the ACID guarantees of a traditional database system.

https://en.wikipedia.org/wiki/NewSQL

MPP (massively parallel processing)

• that was the time when Hadoop was not an option for most of the companies, especially for the enterprises that ask for stable and mature platforms. At that very moment the choice was very simple: when your analytical database grow beyond 5-7 terabytes in size you just initiate an MPP migration project and move to one of the proven enterprise MPP solutions

No one heard about the “unstructured” data – if you got to analyze logs just parse them with Perl/Python/Java/C++ and load into you analytical DBMS. And no one heard about high velocity data – simply use traditional OLTP RDBMS for frequent updates and chunk them for insertion into the analytical DWH

And as Hadoop became more and more popular, MPP databases entered their descent.
So the question regarding “whether I should choose MPP solution or Hadoop-based solution?”
Many of the vendors are positioning Hadoop as a replacement of the traditional data warehouse, meaning by this the replacement of the MPP solutions.
Some of them are more conservative in the messaging and pushing the Data Lake / Data Hub concept, when Hadoop and MPP leave beside each other and integrating together in a single solution.

MPP stands for Massive Parallel Processing, this is the approach in grid computing when all the separate nodes of your grid are participating in the coordinated computations.
MPP DBMSs are the database management systems built on top of this approach. In these systems each query you are staring is split into a set of coordinated processes executed by the nodes of your MPP grid in parallel, splitting the computations the way they are running times faster than in traditional SMP RDBMS systems.
One additional advantage that this architecture delivers to you is the scalability, because you can easily scale the grid by adding new nodes into it. To be able to handle huge amounts of data, the data in these solutions is usually split between nodes (sharded) the way that each node processes only its local data.
This further speeds up the processing of the data, because using shared storage for this kind of design would be a huge overkill – more complex, more expensive, less scalable, higher network utilization, less parallelism. This is why most of the MPP DBMS solutions are shared-nothing and work on DAS storage or the set of storage shelves shared between small groups of servers.

Hadoop storage technology is built on a completely different approach. Instead of sharding the data based on some kind of a key, it chunks the data into blocks of a fixed (configurable) size and splits them between the nodes. The chunks are big and they are read-only as well as the overall filesystem (HDFS).

The chunks are big and they are read-only as well as the overall filesystem (HDFS). To put it simple, loading small 100-row table into MPP would cause the engine to shard the data based on the key of your table, this way in a big enough cluster there is a huge probability that each of the nodes will store only one row. In contrast, in HDFS the whole small table would be written in a single block, which would be represented as a single file on the datanode’s filesystems.

what about the cluster resource management? In contrast to MPP design, Hadoop resource manager (YARN) is giving you more fine-grained resource management – compared to MPP, the MapReduce jobs does not require all its computational tasks to run in parallel, so you can even process a huge amounts of data within a set of tasks running on a single node if the other part of your cluster is completely utilized. It also has a series of nice features like extensibility, support for long-living containers and so on. But in fact it is slower than MPP resource manager and sometimes not that good in managing concurrency.

you can conclude why Hadoop cannot be used as a complete replacement of the traditional enterprise data warehouse, but it can be used as an engine for processing huge amounts data in a distributed way and getting important insights from your data.

https://0x0fff.com/hadoop-vs-mpp/

• Massively Parallel Processing (MPP) Database on Hadoop

In Massively Parallel Processing (MPP) databases data is partitioned across multiple servers or nodes with each server/node having memory/processors to process data locally. All communication is via a network interconnect — there is no disk-level sharing or contention to be concerned with (i.e. it is a ‘shared-nothing’ architecture).
http://www.grroups.com/blog/massively-parallel-processing-mpp-database-on-hadoop

• In Massively Parallel Processing (MPP) databases data is partitioned across multiple servers or nodes with each server/node having memory/processors to process data locally. All communication is via a network interconnect — there is no disk-level sharing or

contention to be concerned with (i.e. it is a ‘shared-nothing’ architecture)
https://dwarehouse.wordpress.com/2012/12/28/introduction-to-massively-parallel-processing-mpp-database/

• In computing, massively parallel refers to the use of a large number of processors (or separate computers) to perform a set of coordinated computations in parallel (simultaneously).In one approach, e.g., in grid computing the processing power of a large number of computers in distributed, diverse administrative domains, is opportunistically used whenever a computer is available.An example is BOINC, a volunteer-based, opportunistic grid system, whereby the grid provides power only on a best effort basis.In another approach, a large number of processors are used in close proximity to each other, e.g., in a computer cluster. In such a centralized system the speed and flexibility of the interconnect becomes very important, and modern supercomputers have used various approaches ranging from enhanced Infiniband systems to three-dimensional torus interconnects

https://en.wikipedia.org/wiki/Massively_parallel

• MPP (massively parallel processing) is the coordinated processing of a program by multiple processor s that work on different parts of the program, with each processor using its own operating system and memory . An MPP system is considered better than a symmetrically parallel system ( SMP ) for applications that allow a number of databases to be searched in parallel. These include decision support system and data warehouse applications

https://whatis.techtarget.com/definition/MPP-massively-parallel-processing

• What is a data lake?

A data lake is a centralized repository that allows you to store all your structured and unstructured data at any scale. You can store your data as-is, without having to first structure the data, and run different types of analytics—from dashboards and visualizations to big data processing, real-time analytics, and machine learning to guide better decisions.

Data Lakes compared to Data Warehouses – two different approaches

A data warehouse is a database optimized to analyze relational data coming from transactional systems and line of business applications. The data structure, and schema are defined in advance to optimize for fast SQL queries, where the results are typically used for operational reporting and analysis. Data is cleaned, enriched, and transformed so it can act as the “single source of truth” that users can trust.

A data lake is different, because it stores relational data from line of business applications, and non-relational data from mobile apps, IoT devices, and social media. The structure of the data or schema is not defined when data is captured. This means you can store all of your data without careful design or the need to know what questions you might need answers for in the future. Different types of analytics on your data like SQL queries, big data analytics, full text search, real-time analytics, and machine learning can be used to uncover insights

https://aws.amazon.com/big-data/datalakes-and-analytics/what-is-a-data-lake/