Monday, December 26, 2016

HADAMS - Hadoop Automated Deployment And Management Stack - Part II

==============

Here is the First Look of TreeBeard.
This will be available for VMWare & Docker to start with.
It will also act as a base for HADAMS Deployment.
The VMWare Edition can be downloaded from Here.
VMPlayer is available Here

For the Simulation, instead of provisioning real systems on the cloud or elsewhere, we will set up TreeBeard VMs on top of Laptops running PrathamOS Aragorn Edition.Also, to start with we will merge the components into lesser systems then a Typical Production Setup.The first requirement is to set up the HA Cluster.Hence those systems would be provisioned initially...

Read Below as HostName (IP) (username - password) 
  •  DataNode + HBase Region Server
    • HADAMS-DN1-RS1 (192.168.1.2) (hadams - dn1rs1)
    • HADAMS-DN2-RS2 (192.168.1.3) (hadams - dn2rs2)
    • HADAMS-DN3-RS3 (192.168.1.4) (hadams - dn3rs3)
    • HADAMS-DN4-RS4 (192.168.1.5) (hadams - dn4rs4)
  • NameNode-I + ZooKeeper-I + HBase Master + Phoenix
    • HADAMS-NN1-HA-ZK1-HBM-PHN (192.168.1.6) (hadams - nn1zk1)
  • NameNode-II + ZooKeeper-II + Resource Manager-II
    • HADAMS-NN2-HA-ZK2-RM2-HA (192.168.1.7) (hadams - nn2zk2)
  • ZooKeeper-III + Resource Manager-I
    • HADAMS-ZK3-RM1-HA (192.168.1.8) (hadams - zk3rm1)
  • Pig + Hive + Tez + MySql + HCatalog + Oozie
    • HADAMS-PG-HV-TZ-MSQL-HCTLG-OZ (192.168.1.9) (hadams - mprdce)
  • Talend + Kettle + Ambari + Nagios
    • ARAGORN-HADAMS-ETL-SETUP-MONITOR (192.168.1.10) (root - pratham)

NEXT STEPS
  • Set Up VMs.Any other distro will also be fine...just need to ensure that the partitioning is ext3, which is recommended for Hadoop Installation in Production.TreeBeard comes default with the same.  
  • Set Up Instance Identity i.e hostname,ip,gateway,netmask,username,password.
  • Password is for physical login.SSH Login with password is disabled for security purposes.we will set up .pem file for ssh and .ppk for putty.
  • Enable NFS Sharing.
  • Install Oracle Java.
  • Time Syncronization with NTP.
Once above steps have been carried out for all above required instances, we will set up password less ssh between the nodes and proceed with stack installation.

So Let's Begin...

Setting up 192.168.1.10 is simple.Download Aragorn & Follow Installation Guide for system, laptop or desktop as may be the choice.Then change settings as per below.



















Java is already installed and ssh login for this system is not required.Next, open a terminal and run the below commands to install other dependencies.
  1. setupapt 
  2. apt-get install ntp ntpdate ssh rsync sshpass jq nfs-kernel-server nfs-common
A good article for setting up NFS can be found here.Follow it to ensure that var/nfsshare is available on .10, to be mountable, as per requirement anywhere in the network.

Type ntpq -p to see servers system is syncing with.Type date -R to see if the time changed.The time should be synced in a next minute.


To Be Continued...

No comments:

Post a Comment