.. _data_lake_blueprint:

Data Lake
"""""""""

Diagram
''''''''

.. graphviz::

  digraph "arch" {
   rankdir="LR"
   compound=true;
   nodesep=0.6;
   ranksep=1.7;

   subgraph cluster_elastic {
        style="filled,solid";
        color=black;
        fillcolor=lightblue;
        "elastic1"[label="elastic1"];
        "elastic2"[label="elastic2"];
        "elastic3"[label="elastic3"];
        label="Datalake : Elastic cluster";

        subgraph cluster_elastic3 {
          label="Dedicated Elastic\nWorker Hypervisor"
          fillcolor=gold;
          elastic3 [
            style="filled,dashed,rounded"
            shape=box
            fillcolor=white
            color=black
          ]
          elastic3_disks[
              style="filled,solid,rounded"
              shape=component
              fillcolor=grey
              color=black
              label=disks
            ]
        }

        subgraph cluster_elastic2 {
          label="Dedicated Elastic\nWorker Hypervisor"
          fillcolor=gold;
          elastic2 [
            style="filled,dashed,rounded"
            shape=box
            fillcolor=white
            color=black
          ]
          elastic2_disks[
              style="filled,solid,rounded"
              shape=component
              fillcolor=grey
              color=black
              label=disks
            ]
        }

        subgraph cluster_elastic1 {
          label="Dedicated Elastic\nWorker Hypervisor"
          fillcolor=gold;
          elastic1 [
            style="filled,dashed,rounded"
            shape=box
            fillcolor=white
            color=black
          ]
            elastic1_disks[
              style="filled,solid,rounded"
              shape=component
              fillcolor=grey
              color=black
              label=disks
            ]
        }
     }

    "kibana"[
       shape=box
       label="Kibana"
       style=rounded
    ];

    "etl"[
       shape=box
       label="ETL Service\n(Logstash for example)"
       style=rounded
    ];

    "command_line"[
       shape=box
       label="Command Line"
       style=rounded
     ];

    "admins"[
       shape=circle
       label="Nice\nAdmins"
       style=rounded
    ];

    "any_other_component_etl"[
       shape=box
       label="Any other component\nsending to the ETL service\nfor transformation"
       style=rounded
     ];

    "any_other_component_direct"[
       shape=box
       label="Any other component\nsending directly in the data lake"
       style=rounded
     ];

   subgraph cluster_client {
    label="Client";
    "client_node"[label="rsyslog client\nnode"];
   }
   subgraph cluster_mgmtclient {
    label="Management node";
    "client_mgt"[label="rsyslog client\nmgt"];
   }
   subgraph cluster_server1 {
    label="Islet Hypervisor";
    "server_islet"[label="rsyslog"];
   }
   subgraph cluster_server3 {
    label="Worker Hypervisor";
    "server_mgt"[label="rsyslog"];
   }

     elastic1 -> elastic1_disks[color=blue]
     elastic2 -> elastic2_disks[color=blue]
     elastic3 -> elastic3_disks[color=blue]
     client_node -> server_islet [color=red];
     client_mgt -> server_mgt [color=red];
     server_islet -> etl [color=red];
     server_mgt -> etl [color=red];
     etl -> elastic2[color=red,lhead=cluster_elastic]
     elastic3 -> kibana[color=green,ltail=cluster_elastic]
     command_line -> elastic3[color=green,lhead=cluster_elastic]
     any_other_component_etl -> etl[color=red]
     any_other_component_direct -> elastic2[color=red,lhead=cluster_elastic]
     admins -> kibana
     admins -> command_line

  { rank = sink;
    Legend [shape=none, margin=0, label=<
    <TABLE BORDER="0" CELLBORDER="1" CELLSPACING="0" CELLPADDING="4">
     <TR>
      <TD COLSPAN="2"><B>Legend</B></TD>
     </TR>
     <TR>
      <TD>Store to disk</TD>
      <TD BGCOLOR="BLUE"></TD>
     </TR>
     <TR>
      <TD>Read / Query / Pull</TD>
      <TD BGCOLOR="GREEN"></TD>
     </TR>
     <TR>
      <TD>Write / Send / Push</TD>
      <TD BGCOLOR="RED"></TD>
     </TR>
    </TABLE>
   >];
  }

 }


Description
'''''''''''

The Data lake service is built around the following basic ideas :

- All the nodes already forward their syslog messages their rsyslog islet servers.

- Those rsyslog servers, in turn, write those file to the gluster FS.

Let's sieze this opportunity to also forward them to a data lake service for the following features :

- Easy log searching via different means :

  - Web Front End (Kibana)

  - Command line (lstail, to be installed)

  Logs can be queried via several criterias : node names, syslog types, severity ... but also wildcarded words or even regexes. It can also be filtered based on log time ...

- Alerting based text queries in syslog (has to be setup by the admins)

- Metrics based on syslogs

- Able to store large amount of data

Technical aspects
'''''''''''''''''

The chosen technology for this architecture is the OSS version of ElasticSearch. Elastic nodes run in VMs hosted by dedicated Worker servers. If necessary, Elastic can be setup to run on the bare-metal. Benchmarks have to be performed in order to appreciate the performance overhead induced by the use of VMs. Performance may greatly vary depending on the topology of data ingested by the Elasticserach server as well as the type of queries they receive.

It is preferable those worker use ssd disks in order to maximize disk throughput and overall Elasticsearch performance.

With the amount of data growing, the Elasticsearch cluster performance might decrease, be it for ingestion or query requests.
In this case, it is possible to add more nodes in order to spread the load accross more nodes.

The general rules of thumb for ElasticSearch sizing is described here : https://www.elastic.co/guide/en/elasticsearch/reference/7.10/size-your-shards.html

It is up to the administrators to properly adapt the cluster tuning to the usecase(s)

.. todo::  The :term:`ETL` service will be described in this blueprint

.. raw:: latex

  \clearpage