AWS Cloudformation : Intrinsic Functions

AWS Cloudformation templates have Intrinsic functions to use upon on Resources such as properties and metadata attributes

With reference to  http://fortycloud.com/using-amazon-cloudformation-service-to-set-up-your-virtual-private-cloud/, here are few of them

String Handling Functions

Fn::Base64 - Returns the Base64 representation of the input string. This function is typically used to pass encoded data to Amazon EC2 instances by way of the UserData property.

Fn::Join - Appends a set of values into a single value, separated by the specified delimiter. If a delimiter is the empty string, the set of values are concatenated with no delimiter.

Functions for Managing data and variables inside the Template

Fn::FindInMap Returns the value corresponding to keys in a two-level map that is declared in the Mappings section.

Fn::GetAtt Returns the value of an attribute from a resource in the template.

Ref Returns the value of the specified parameter or resource.

Region Selection Function

Fn::GetAZs Returns an array that lists Availability Zones for a specified region.

Chef 12 Installation in CentOS

Chef has 3 components:

• Chef Server (192.168.56.101) - CentOS 6.4
• Chef Workstation (192.168.56.102) - CentOS 7.0
• Chef Client(Node) (192.168.56.103) - CentOS 6.4

Configure hostnames

In CentOS 6.4, set the hostname in /etc/sysconfig/network file as follows

HOSTNAME=chef-server    (in chef server 192.168.56.101)

HOSTNAME=chef-client     (in chef client 192.168.56.103)

In CentOS 7.0, set the hostname

hostnamectl set-hostname chef-workstation (192.168.56.102)

Then in /etc/hosts file in each of the above hosts make an entry as follows

192.168.56.101 chef-server chef-server.localdomain

192.168.56.102 chef-workstation chef-workstation chef-workstation

192.168.56.103 chef-client chef-client chef-client

Chef Server Installation

# wget https://web-dl.packagecloud.io/chef/stable/packages/el/6/chef-server-core-12.2.0-1.el6.x86_64.rpm

# rpm -ivh chef-server-core-12.2.0-1.el6.x86_64.rpm

# chef-server-ctl reconfigure

# chef-server-ctl test

# mkdir -p /etc/chef-server/

Create Admin User

Syntax for creating a chef user account

chef-server-ctl user-create user_name first_name last_name email password --filename FILE_NAME

An RSA private key is generated automatically. This is the user’s private key and should be saved to a safe location. The --filename option will save the RSA private key to a specified path. 

Iam creating a chef user "chefadmin" whose key is chefadmin.pem

chef-server-ctl user-create chefadmin ChefUser Admin chefadmin@chef-server.com 1q2w3e4r --filename /etc/chef-server/chefadmin.pem

Create an Org

Syntax for creating a chef org

chef-server-ctl org-create short_name "full_organization_name" --association_user user_name --filename ORGANIZATION-validator.pem

The --association_user option will associate the user_name with the admins security group on the Chef server.

An RSA private key is generated automatically. This is the chef-validator key and should be saved to a safe location. The --filename option will save the RSA private key to a specified path

Iam creating a org chefserver whose key is chefserver-validator.pem using the user chefadmin created earlier

chef-server-ctl org-create chefserver ChefServer --association_user chefadmin --filename /etc/chef-server/chefserver-validator.pem

Chef workstation setup

As root user run the following command

# curl -L https://www.opscode.com/chef/install.sh | bash

2) When the installation is finished enter the chef-client command to verify that the chef-client was installed:

# chef-client -v

3) Under a normal user, I will  create the “.chef” directory under the user's home directory /home/xyz/.chef, where xyz is the username

The .chef directory is used to store three files:

• knife.rb
• ORGANIZATION-validator.pem
• USER.pem

The *.pem keys are the ones generated in Chef Server for User and Organization.

Need to copy those keys from Chef server to Chef Workstation

$ cd /home/xyz/

Copying the User key from Chef Server to Chef workstation

$ scp root@chef-server:/etc/chef-server/chefadmin.pem ~/.chef/

Copying the Organization key from Chef Server to Chef workstation

$ scp root@chef-server:/etc/chef-server/chefserver-validator.pem ~/.chef/

Configure knife configuration file  ~/.chef/knife.rb file
log_level                :info
log_location             STDOUT
node_name                'chefadmin'
client_key               '/home/xyz/.chef/chefadmin.pem'
validation_client_name   'chefserver-validator'      
validation_key           '/home/xyz/.chef/chefserver-validator.pem'  
chef_server_url          'https://chef-server:443/organizations/chefserver'
syntax_check_cache_path  '/home/xyz/chef-repo/.chef/syntax_check_cache'

Run knife ssl fetch to trust the server’s self-signed cert.

knife client list should now show you the name of your validator, which in this case is:

chefserver-validator

knife user list

chefadmin

BootStraping Chef Client

We will install Chef client software in chef-client machine from chef-workstation machine

Bootstrapping a node installs the chef-client and validates the node, allowing it to read from the Chef server.

1) From Chef workstation, bootstrap the chef client node by using the chef client node’s root user

   knife bootstrap <Chef Client IP> -x root -P password --node-name <nodename>

   <nodename> is optional. If not specified it will take the hosname of Chef client node as nodename

    knife bootstrap <Chef Client IP or hostname>

    $ knife bootstrap chef-client (or)  knife bootstrap 192.168.56.103

2) Confirm that the node has been bootstrapped by listing the nodes in Chef Workstation by running the command

 $ knife node list 

Reference

• https://www.digitalocean.com/community/tutorials/how-to-create-simple-chef-cookbooks-to-manage-infrastructure-on-ubuntu
• https://www.linode.com/docs/applications/chef/setting-up-chef-ubuntu-14-04

Chef 12 Workstation : Response: missing read permission

While setting up Chef Workstation, after configuring ~/.chef/knife.rb file, tried validating the Chef Workstation with Chef Server by running the command

[chef-workstation .chef]$ knife user list

ERROR: You authenticated successfully to https://chef-server:443 as chefadmin but you are not authorized for this action

Response:  missing read permission

Upon analyzing the cause, it was figured out that in the file ~/.chef/knife.rb in Chef Worsktation, the entry for chef_server_url was wrongly specified as

chef_server_url          'https://chef-server:443/'  - Wrong

From Chef 12, this should be specified as

chef_server_url          'https://chef-server:443/organizations/xxxx' - Correct

where, xxxx - Name of the Organization created in Chef Server

Ruby Path variable $LOAD_PATH

In Ruby, to find the list of directories which are searched by load and require methods we can use the global variable $LOAD_PATH

#!/usr/bin/env ruby

p $LOAD_PATH
p $:

$: is a short synonym for $LOAD_PATH name

How to find boto version installed?

To find the version of boto installed, run the following program

#!/usr/bin/python

import boto

print boto.Version

When I ran the above program, I got the version of boto installed in my system as 2.32.1

AWS: SSH ProxyCommand to login directly to private instance

In AWS, to ssh into the private server instance, we need to first ssh into bastion host first. Only from bastion host we shall be able to login into private server instances. Hence we need to store our ssh private key into the bastion host to be able to login to private instances.

But storing ssh private key in bastion host is not a good practise. To overcome that, there are two possibilities

1) Use ssh-agent for forwarding keys through bastion host

2) Use ssh ProxyCommand

Let me explain the later option of using ssh ProxyCommand to login to AWS private instance by tunneling through bastion host.

From our localhost(desktop client), we need to

1) SSH into our bastion host

2) Run netcat command on the bastion host to open a connection to the remote host(private aws instance)

3) Connect to the remote host(private aws instance) through the netcat tunnel from the local desktop without having to store the private ssh key in the bastion host.

OpenSSH 5.4 and above have netcat built in. So in our local desktop, we need to configure ssh client configuration ~/.ssh/config as below

Host privateecinstance

     Hostname <aws_private_instance_ip>

     User ec2-user                                              #Username to ssh into private ec2 instance

     ProxyCommand ssh -W %h:%p ec2-user@<bastion-host-ip>  2> /dev/null

Now we can login to private ec2 instance from our local desktop as follows

ssh privateecinstance

Fluentd, ElasticSearch, Kibana Installation in CentOS 7

To aggregate logs in a single place and have an integrated view of aggregated logs through a UI, people normally use ELK stack.

• Fluentd - For aggregating logs in a single server
• Elasticsearch - For Indexing the aggregated logs
• Kibana - GUI for viewing the logs

I will install all three, Fluentd, Elasticsearch, Kiabana in single host

Fluentd

Logs are streams - no beginning or end. We need to send logs from all the hosts to the Elasticsearch server for indexing. For streaming logs to a centralized server, we have various tools like Fluentd, LogStash, Flume, Scribe. Here Iam using Fluentd. 

Installation

Fluentd is available as td-agent or fluentd package. Here Iam using td-agent.

Difference between fluentd and td-agent is listed in

           http://www.fluentd.org/faqs

curl -L https://td-toolbelt.herokuapp.com/sh/install-redhat-td-agent2.sh | sh

Configuration

Fluentd configuration file is /etc/td-agent/td-agent.conf

Configure Fluentd to aggregate rsyslog messages to elasticsearch as follows

# collect the dmesg output

<source>

  type syslog

  port 42185

  tag syslog

</source>

<match syslog.**> 

  type elasticsearch 

  logstash_format true         #Kibana understands only logstash format

  flush_interval 10s # for testing 

</match> 

Start fluentd(td-agent)

#Check the status of td-agent service
/etc/init.d/td-agent status

#To enable td-agent to start on boot automatically
/etc/init.d/td-agent enable

#To start the td-agent service

/etc/init.d/td-agent start

#In CentOS 7 using the systemctl command to check status, start, stop td-agent service
systemctl status td-agent
systemctl start td-agent
systemctl stop td-agent

Next we will install Elasticsearch. But Elasticsearch needs Java. So we will start with Java installation first

fluentd(td-agent) log file and pid path

Log file   : /var/log/td-agent/td-agent.log

PID path : /var/run/td-agent/td-agent.pid

Java8 Installation 

cd /opt

wget --no-cookies --no-check-certificate --header "Cookie: 

gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u45-b14/jdk-8u45-linux-x64.tar.gz" 

tar xzf jdk-8u45-linux-x64.tar.gz 

alternatives --install /usr/bin/java java /opt/jdk1.8.0_45/bin/java 2                                                           alternatives --config java                                                                                                                            alternatives --install /usr/bin/jar jar /opt/jdk1.8.0_45/bin/jar 2                                                                 alternatives --install /usr/bin/javac javac /opt/jdk1.8.0_45/bin/javac 2                                                     alternatives --set jar /opt/jdk1.8.0_45/bin/jar                                                                                             alternatives --set javac /opt/jdk1.8.0_45/bin/javac      

sh -c "echo export JAVA_HOME=/opt/jdk1.8.0_45 >> /etc/environment"

sh -c "echo export JRE_HOME=/opt/jdk1.8.0_45/jre >> /etc/environment"  

sh -c "echo export PATH=$PATH:/opt/jdk1.8.0_45/bin:/opt/jdk1.8.0_45/jre/bin >> /etc/environment"

cat /etc/environment

java -version

Elasticsearch

yum repo setup

cd /opt/

rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

cd /etc/yum.repos.d/

vi elasticsearch.repo

   name=Elasticsearch repository for 1.6.x packages

   baseurl=http://packages.elastic.co/elasticsearch/1.6/centos

   gpgcheck=1

   gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch

   enabled=1

Installation

yum install elasticsearch

#Check the status of elasticsearch service

systemctl status elasticsearch

#Enable the elasticsearch service to be started on boot

systemctl enable elasticsearch

#Start the elasticsearch service

systemctl start elasticsearch

Status Check

curl http://localhost:9200/

tail -f /var/log/elasticsearch/elasticsearch.log

Elasticsearch system configuration setting changes

https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html

Elasticsearch Common Query commands

Any elasticsearch info is organized as

curl 'localhost:9200/<index>/<type>/<id>/'

#To get a list of indices

curl 'localhost:9200/_cat/indices?v'

In our case, the indices will be of type logstash-<yyyy.mm.dd>, as Fluentd is sending it in logstash format

#To check for data under a index

curl 'localhost:9200/logstash-<yyyy.mm.dd>/_search?pretty=true'

#To query for data under a index and a type(it is "fluentd" in our case)

curl 'localhost:9200/logstash-<yyyy.mm.dd>/fluentd/_search?pretty=true'

#To know if there is data available for a time period in elasticsearch index

curl 'localhost:9200/logstash-<yyyy.mm.dd>/fluentd/_search?q="00:00"&pretty=true'

#Get cluster health

curl 'localhost:9200/_cluster/health'

#To get the health of an index, for example say for the index logstash-2015.07.01

curl -XGET 'http://localhost:9200/_cluster/health/logstash-2015.07.01'

Kibana 4

Installation

cd /opt

wget https://download.elastic.co/kibana/kibana/kibana-4.1.0-linux-x64.tar.gz

tar xzvf kibana-4.1.0-linux-x64.tar.gz

mv kibana-4.1.0-linux-x64 /opt/kibana4

#Enable PID file for Kibana, this is required to create a systemd init file.

sed -i 's/#pid_file/pid_file/g' /opt/kibana4/config/kibana.yml

Start/Stop

Kibana4 service can be started by running /opt/kibana4/bin/kibana, 

# kibana4 start/stop systemd script

 vi /etc/systemd/system/kibana4.service

[Unit]

Description=Kibana 4 Web Interface

After=elasticsearch.service

After=td-agent.service

[Service]

ExecStartPre=rm -rf /var/run/kibana.pid

ExecStart=/opt/kibana4/bin/kibana/

ExecReload=kill -9 $(cat /var/run/kibana.pid) && rm -rf /var/run/kibana.pid && /opt/kibana4/bin/kibana/

ExecStop=kill -9 $(cat /var/run/kibana.pid)

[Install]

WantedBy=multi-user.target

# Start and enable kibana to start automatically at system startup.

systemctl start kibana4.service

systemctl enable kibana4.service

Kibana Portal Access

http://<kibana_server-ip-address>:5601/