Cybot Analytical Engine Installation

Cybot Analytical Engine Installation

Cybot Analytical Engine Installation

 This guide will lead you through the process of deploying a Cybot Analytical Engine.

Downloading the OVA

  1. The download link for the ova will be shared by the ActiveBytes Support team.
  2. Download the OVA and proceed to import the OVA using your preferred hypervisor.

Importing the OVA – Hypervisor

Deploy a Virtual Machine from an OVA File. Allocate resources according to the VM as per the hardware prerequisite.

Configuration.

Once the installation is complete, login to the workstation with the user name and password which will be provided by the CyBot support team.

  1. Power on the machine.
  2. Click on the machine.
  3. Enter the username and password to login the Virtual Machine provided by Cybot Support

 

Setting the IP for Virtual Machine

In-order to SSH into the VM via any SSH Client, assign an IP address to the VM.

To Configure IP address, you can navigate to "Configure IP in Virtual Machines". This document will guide you to configure IP address in the Virtual Machine.   

Setting up the host name

Once the IP address is assigned for the VM, configure the preferred host name for the Cybot Analytical Engine .

  1.  Check current host-name
            >>> hostnamectl
  1. Change hostname
            >>> sudo hostnamectl set–hostname <newhostname>
  1.    Apply the changes
            >>> sudo reboot (this will reboot the machine)
  1. Check the hostname again using
            >>> hostnamectl

Partitioning / Mounting the Hard disk.

The ideal design for setting up a Cybot Analytical Engine  will include 2 hard disk, one dedicated for the OS and the other for the data which is stored in Cybot Analytical Engine.

Mounting steps

Check disk partition
>>> lsblk

To create a mounting path for the hard disk, follow the below steps

>>> mkfs.ext4 <hard disk name>
Sample: mkfs.ext4 /dev/sdb

>>> mkdir /mnt/<provide a folder name>
Sample mkdir /mnt/cyb_datalake

>>> mount  <hard disk name>
Sample: mount /dev/sdb   /mnt/cyb_datalake


Recheck the disk partition and you should be able to see the mount point for the hard disk:
>>> lsblk

                                    

Cybot Analytical Engine configuration

Cybot Analytical Engine 

This is the big data analytical engine for CyBot. To configure any changes to modify the IP address or port you may change in the configuration file as shown below.

>>>> nano /home/cybot/opensearch-2.8.0/config/opensearch.yml

                        

  1.  To change the IP address or port where the Cybot Analytical engine must run, you need to provide the IP address and port for the respective parameters                  

network.host: localhost (Recommended)

network.port: 9200 (Recommended)


                          
               
  1.  Add the newly added mount path for the Cybot Analytical engine for storing the data and logs under the following parameters.

path.data: /mnt/<path to mount point>/cybot/data

path.log: /mnt/<path to mount point>/cybot/log

 



  1. Each time a configuration is changed in the opensearch.yml file, you will have to restart the Cybot Analytical engine service for the changes to be reflected.
  2. Before running as a service if you want to run manually to check if your configurations are working, please use the following commands:

>>>> su cybot

>>>> cd /home/cybot/opensearch-2.8.0/                            

>>>> bin/opensearch

                        

            







  1. For restarting the service, please find the following command:

                              >>>> systemctl restart datalake-engine

Datalake Engine Nodes can be Single or Multi-Nodes as per requirement, Above setup shows how to setup a single node cluster. To learn how to setup a Multi-Node Setup find the below steps.
To setup Multi-Node Datalake Cluster, follow the steps below (steps 1-5 can be followed as mentioned above):
  1. Downloading the OVA
  2. Importing OVA
  3. Setting the IP for Virtual Machine
Once the machine is setup, now enter the configuration file

>>>> nano /home/cybot/opensearch-2.8.0/config/opensearch.yml

                        

  1.  To change the IP address or port where the Cybot Analytical engine must run, you need to provide the IP address and port for the respective parameters                  

network.host: localhost (Recommended)

network.port: 9200 (Recommended)


                          
               
  1.  Add the newly added mount path for the Cybot Analytical engine for storing the data and logs under the following parameters.

path.data: /mnt/<path to mount point>/cybot/data

path.log: /mnt/<path to mount point>/cybot/log

 



Now provide appropriate roles and tiers for the datalake respectively
Tiers that Datalake supports:
  1. hot
  2. warm
  3. cold
  4. frozen

Roles that Datalake Supports:

  1. cluster_manager
  2. data
  3. ingest
In the configuration file, add the appropriate roles and tiers repectively for each Datalake machine as shown below:
This shows a Datalake machine that will be acting as a Hot Node and the roles assigned are as a cluster manager and to act as a data Node.


This shows a Datalake machine that will be acting as a Warm Node and the roles assigned is only to act as a data Node.


This shows a Datalake machine that will be acting as a Cold Node and the roles assigned is only to act as a data Node.


Now to connect all the datalake machines together as a cluster, provide the IP's of each Datalake machine in the discovery.seed_hosts as shown below. All the machines that needs to be connected to the cluster has to be mentioned in the "discovery.seed_hosts" seperated by comas. And the Datalake machine that will be acting as the cluster Manager needs to be mentioned in the "cluster.initial_cluster_manager_nodes"
for e.g  discovery.seed_hosts: ["0.0.0.0", "1.1.1.1", "2.2.2.2"]
             cluster.initial_cluster_manager_nodes : ["0.0.0.0"]


Now restart datalake engine services in all the Node machines as shown below. 
      >>>> systemctl restart datalake-engine 

Confirm the clusters have been connected by using the below URL in the browser.
https://<cluster-manager-ip>:<port_number>/_cluster/health



You can confirm your cluster nodes by checking the "number_of_nodes" and "number_of_data_nodes".

Now you have now Successfully completed Multi Cluster Datalake installation.


Cybot Analytical engine - UI 

This is the UI for your Cybot Analytical Engine. To configure any changes to modify the IP address or port you may change in the configuration file as shown below. 

>>>> nano /home/cybot/opensearch-dashboards-2.8.0/config/opensearch-dashboards.yml 


 

 

To change the IP address or port where the Cybot Analytical engine engine must run, you need to provide the IP address and port for the respective parameters.                    

server.host: x.x.x.x 

server.port: 5601 

 

 

 

  • Each time a configuration is changed in the opensearch-dashboards.yml file, you will have to restart the Cybot Analytical engine service for the changes to be reflected. 

  • Before running as a service if you want to run manually to check if your configurations are working, please use the following commands: 

>>>> cd /home/cybot/opensearch-dashboards-2.8.0/ 

>>>> bin/opensearch-dashboards 

 

For restarting the service, please find the following command: 

>>>> systemctl restart datalake-ui.service

 


Services for Cybot Analytical engine 

Cybot Analytical Engine Service (datalake-engine.service):

Within our Cybot Analytical Engine ecosystem, the "datalake-engine.service" plays a pivotal role in enabling robust data storage, real-time indexing, and search capabilities. This service empowers users to access immediate results as data is ingested, enhancing the efficiency and responsiveness of our data analytics.

Service Health Monitoring:

To ensure the continuous and optimal performance of the "datalake-engine.service," you can monitor its status and health by executing the following command:

“systemctl status datalake-engine.service”

This command will provide you with valuable insights into the service's status, allowing you to promptly identify any issues or anomalies.

                                    

Service Restart:

In the event of service failure or if the need arises to restart the "datalake-engine.service" for any reason, you can initiate the restart process using the following command:

systemctl restart datalake-engine.service

This command will gracefully restart the service, and subsequent checks of the service status will confirm its successful restoration to operational status.

Validating via CLI

Validating through the cli ensure that the services are running fine locally (in the machine)

Cybot Analytical engine :
>>> curl -i -k -u <username>:<password> https://<machine_IP>:9201




Cybot Analytical engine User Interface Service (datalake-ui.service): 

The "datalake-ui.service" is a crucial component in our Cybot Analytical Engine infrastructure, responsible for running the User Interface (UI) for the Cybot Analytical Engine. This UI empowers users to create interactive and visually appealing dashboards, facilitating data exploration and insights. 


Service Health Monitoring: 

To monitor the status and health of the "datalake-ui.service," you can use the following command: 

systemctl status datalake-ui.service 


This command provides a snapshot of the service's status, enabling you to quickly assess its operational health. 

 


Service Restart: 

In the event of a service failure or the need for a restart, you can initiate the process with the following command: 

      >>>> systemctl restart datalake-ui.service 


Executing this command will gracefully restart the service. Subsequently, you can verify the service's status to ensure that it has been successfully reinstated. 

The Cybot Analytical Engine User Interface Service empowers users to harness the potential of their data through interactive and visually engaging dashboards, fostering data-driven insights and decision-making. 


 

Starting up the Cybot Analytical Engine services (data lake, dashboard, Load balancer) 

Use the commands below to start the services. 

      >>>> systemctl restart datalake.service / systemctl status datalake.service
      >>>> systemctl restart datalake-ui.service / systemctl status datalake-ui.service 
      >>>> systemctl restart nginx / systemctl status nginx  



If the services are getting failed continuously, please try restarting the service after using the following command. 

      >>>> setenforce 0



Now you have restarted all the services required for the Cybot Analytical Engine to be operational, proceed to validate the services using below steps: 


Validation via browser 

Open any browser of preference. 

Go to: https://<machine_IP>:5600 

 


Now you have now Successfully completed installation for the Cybot Analytical Engine.



    • Related Articles

    • Installation Files

      Installation of Cybot is streamlined through pre-configured OVA files. To obtain these files, please get in touch with the Cybot support team. OVA: It's a standard format used in virtualization environments to package and distribute pre-configured ...

    • CYBOT™ Architecture

      This document will offer a deep dive into the intricate components that form the foundation of CYBOT™ architecture. Understanding its architecture is essential for developers, researchers, and enthusiasts looking to harness its potential. Components ...

    • Cybot™ Platform

      CYBOT™ is a one-of-a-kind platform that is custom-built to fit needs of enterprises with critical infrastructure. It acts as a cyber security foundation for organizations that requires deep visibility into their infrastructure with a multilayer ...

    • Cybot™ Log Collector

      Installation steps to deploy Datalake Engine and Datalake-UI Downloading the OVA The download link for the ova will be shared by the ActiveBytes Support team. Download the OVA and proceed to import the OVA using your preferred hypervisor. Importing ...

    • About CYBOT™

      Introduction to CYBOT™ CYBOT™ is a cutting-edge solution developed by ActiveBytes Innovations to address the challenges posed by the overwhelming volume and complexity of logs generated in today's enterprise landscape. This groundbreaking platform ...