How to increase/reduce the verbosity of the Apache httpd error log?

To increase or decrease the information being logged into Apache's error log file, LogLevel directive in httpd.conf file need to be set with any one of the following levels - emerg, alert, crit, error, warn, notice, info, debug

Eg:

LogLevel warn

emerg results in the least information being recorded and debug the most.

The path of the error log file is specified using ErrorLog directive in Apache httpd.conf file. 

The ErrorLog directive sets the name of the file to which the server will log any errors it encounters. If the file-path is not absolute then it is assumed to be relative to the ServerRoot.

Eg:

    ErrorLog logs/error_log       #ErrorLog path relative to ServerRoot

     (or)

   ErrorLog /var/log/httpd/error_log

How to Increase the number of simultaneous/concurrent TCP connections in Red Hat Enterprise Linux 5 and 6 ?

/var/log/messages may show that number of TCP connections has been exhausted and kernel is dropping packets.

To modify the maximum allowed TCP sessions in RHEL 5 and 6, with reference to https://access.redhat.com/site/solutions/94803, here is the solution

In RHEL 5, modify the following parameter

    /proc/sys/net/ipv4/netfilter/ip_conntrack_max

In RHEL 6, modify the following parameter

     /proc/sys/net/ipv4/netfilter/ip_conntrack_max

The increase in TCP session limit will result in more core memory being used.

File system commonly available on Unix and Unix-like operating systems (Linux, FreeBSD, Solaris)

Which file system is commonly available in Unix, Linux, FreeBSD and Solaris?

The virtual filesystem, procfs("process" file system or /proc, where it is usually mounted) is the filesystem commonly available in Unix and Unix like operating systems, namely, Linux, FreeBSD and Solaris.

File types in Linux

Different file types in Linux are

1. Regular file  ( - )
2. Directory ( d )
3. Symbolic Link ( l )
4. Character Special Device ( c )
5. Block Device ( b )
6. Named pipe or Fifo ( p )
7. Socket file ( s )

Apache Connection Establishment and serving requests

Apache server establishes a TCP connection with the client(browser) before serving requests.
Apache puts it's socket into LISTEN state using listen() system call.

TCP three-way handshake between client and server, can be briefed as follows:

1) Client end
    Client sends SYN
    Client moves into SYN SENT state

2) Server end
    Kernel hands SYN to Server
    Kernel sends SYN/ACK to the client
    Server moves to SYN RCVD state
    Connections in SYN RCVD state are added to SYN QUEUE(incomplete connection queue)

3) Client end 
   Client sends ACK

TCP Connection is ESTABLISHED. In the server end, once the ACK packet is received from the client, the connections in the SYN QUEUE(incomplete connection queue) are moved to the ACCEPT QUEUE. Apache will use the accept() system call to consume connections from the ACCEPT QUEUE.

Once the tcp connection is established between client and server, now the first http request data is passed from client to server

In the server end, the kernel passes connection into the server's accept() method.

The Kernel accepts up to /proc/sys/net/ipv4/tcp_max_syn_backlog (default 1024) requests to be in SYN_RCVD state(SYN QUEUE) - incomplete connections. Maximal number of remembered connection requests, which still did not receive an acknowledgement from connecting client. For each SYN requests received, it sends back SYN/ACK and awaits an ACK from the client. If the server, does not receive an ACK from the client, it retries sending SYN/ACK - /proc/sys/net/ipv4/tcp_synack_retries times (default 5 in linux), before dropping the connection. Once all the retries are failed, the client will get a RST packet from the server.

So at server end, we basically deal with two queues in Linux TCP implementation

1. Incomplete connection queue(SYN QUEUE - Connections in SYN RCVD state)
2. Accept Queue(Connections in ESTABLISHED state)

If the Accept Queue is full, then the kernel will impose a limit on the rate at which SYN packets are accepted. If too many SYN packets are received, some of them will be dropped.

It is the job of the application(Apache, in this case) to empty the Accept Queue(completed connection queue) by calling the accept() system call. 

Apache will accept() up to 'MaxClients' simultaneous client connections. The kernel will queue up to lower of either of the following - /proc/sys/net/core/somaxconn (default 128) or 'ListenBackLog' (default 511) -additional established connections. somaxconn is the Limit of socket listen() backlog. Going by the default values of  /proc/sys/net/core/somaxconn (default 128) and 'ListenBackLog' (default 511), the kernel shall queue upto 128 established connections since 128 is lower than 511. Above this limit, the kernel will drop additional SYN packets.

How to limit simultaneous connections per client?

Apache does not have an in-built mechanism to limit simultaneous connections per client. So by using tools such as iptables,it can be achieved.

Reference:
http://www.cyberciti.biz/files/linux-kernel/Documentation/networking/ip-sysctl.txt
http://veithen.github.io/2014/01/01/how-tcp-backlog-works-in-linux.html

Python script for checking if an IP address is valid

To know if an IP address is valid, here is a simple python script. Let us name the script as chkip.py

Pass the IP address to be validated as an argument for this program, like follows
python chkip.py  <IP Address>


#!/usr/bin/python
import re
import sys

def check_ip_validity(ipaddr):
    ValidIpAddressRegex = '^(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$'
    ip_match = re.search(ValidIpAddressRegex,ipaddr)
    if ip_match:
       ip = 1
    else:
       ip = 0
    return ip

def main():
    ip = 0
    if check_ip_validity(sys.argv[1]):
       print "Valid IP"
    else:
       print "Not a valid IP"

if __name__ == '__main__':
    main()

NRPE and NSCA in Nagios

NRPE is used for performing active checks in Nagios
NSCA is used for performing passive checks in Nagios

check_nrpe (from NRPE) is used to initiate an active check from the  Nagios server. check_nrpe is called like a normal plugin by nagios server. check_nrpe initiated in the server, then contacts the nrpe daemon running on a  remote host(a host monitored by Nagios) and asks it to run a pre-configured check command and  returns the results to nagios server.

NSCA is used when a passive check is initiated from a remote host(a host monitored by Nagios server). In the remote host, a  wrapper script for checking a service, is called by cron or some other method,  which in turn  passes the results to the send_nsca  program running in the same remote host. send_nsca contacts the nsca daemon on the nagios server. nsca daemon passes the results of the check to nagios.

Checking SSL certitificate expiry date

SSL certificates, called as digital certificates or X.509 certificates, are used to verify the authenticity of the web server by the browser i.e., is this the server host whom they claim to be(am i connecting to the correct web server). The SSL certificates are authenticated by third party Certificate Authorities.

To view the expiry date of an ssl certificate on a domain, say for example www.facebook.com

$ openssl s_client -connect www.facebook.com:443 | openssl x509 -text | grep -i "Not"
depth=2 C = US, O = "VeriSign, Inc.", OU = Class 3 Public Primary Certification Authority
verify return:1
depth=1 O = VeriSign Trust Network, OU = "VeriSign, Inc.", OU = VeriSign International Server CA - Class 3, OU = www.verisign.com/CPS Incorp.by Ref. LIABILITY LTD.(c)97 VeriSign
verify return:1
depth=0 C = US, ST = California, L = Palo Alto, O = "Facebook, Inc.", CN = *.facebook.com
verify return:1
            Not Before: Jun 21 00:00:00 2012 GMT
            Not After : Dec 31 23:59:59 2013 GMT

To view the expiry date of an ssl certificate(<server>.crt) in the web server where it is stored

The path of the <server>.crt file shall be available in the directive "SSLCertificateFile" in the httpd.conf file

# openssl x509 -noout -in <server>.crt -dates

Eg : openssl x509 -noout -in /etc/httpd/conf/server.crt -dates
notBefore=Sep 18 15:40:02 2012 GMT

notAfter=Sep 18 15:40:02 2013 GMT

Difference between SSL(Secure Sockets Layer) and TLS(Transport Layer Security)

1) Open version of SSL is referred to as the TLS protocol. So TLS is a open protocol while SSL is not.

2) To use SSL with a particular protocol(say, HTTP), it is necessary to have a different port no.(for https, 443) for communication. This is done in order to avoid affecting other clients who communicate on the default clear-channel port(80). So we have port 80 reserved for HTTP and port 443 for SSL over HTTP. Managing multiple requests under both SSL and clear channel may become complex. 

    TLS, however, allows to access both types of traffic(secure and non-secure) over the same default port. No separate port is needed while using TLS.

How SSL protocol works?

SSL/TLS protocol sits in-between the Application Layer(HTTP) and TCP/IP layer. It handles the encryption & decryption for a secure communication between the client and the server. So, SSL is not a part of HTTP and it is a separate layer.

SSL uses an encryption technique called public key cryptography, where the server end of the connection sends the client a public key for encrypting information, which only the server can decrypt with the private key it holds. The client uses the public key to encrypt and send the server it's own key, identifying it uniquely to the server. This prevents man-in-the-middle attack.

In addition to encrypting the connection, we need to ensure that the client is connecting to the right web server, before setting up an SSL connection. The verification of authenticity of the web server is done using Digital certificates issued for the web server, which is authenticated by third party certificate authorities.

The server sends the digital certificate along with a encrypted random data to the client. The client(browser) verifies the certificate before setting up a SSL session. The certificate verification is done as follows

1) The certificate must be able to decrypt the random data that was sent from the server, which is encrypted by the server's private key.
2) The certificate must have been issued by an accepted CA
3) The name on the certificate must match the host name of the server requested by the client.
4) The certificate must not have expired.

If these criteria are met, the browser will then exchange a session key with the server. The session key is a random bit of data that will be used to encrypt the rest of the session.

In Summary:

• The server sends the client its public key and certificate.
• The client checks that the certificate was issued by a trusted party (usually a trusted Certificate Authority) that the certificate is still valid, and that the certificate is related to the contacted site. If the client trusts the server, it sends a message to the server. The server sends back a digitally signed acknowledgement to start an SSL encrypted session.
• The client uses the public key to encrypt a random symmetric encryption key and sends it to the server, along with the encrypted URL required and other encrypted HTTP data.
• The server decrypts the symmetric encryption key using its private key and uses the symmetric key to decrypt the URL and HTTP data.
• The server sends back the requested HTML document and HTTP data that are encrypted with the symmetric key.
• The client decrypts the HTTP data and HTML document using the symmetric key and displays the information.

Is SSL really Secure?

Data sent from browser to the server is managed unencrypted in server memory. If the server does a core dump, that it shall be possible to read  all the data sent by the client to the server.

Reference:
http://stackoverflow.com/questions/6241991/how-exactly-https-ssl-works

facter - Puppet tool to get information about a system/host

facter, a Puppet library and a command-line tool (/usr/bin/facter), gives all the information  about a host in key,value format

1) Install facter in centos
 yum install facter

2) [root@dhcppc3 ~]# facter
architecture => x86_64
boardmanufacturer => Intel Corporation
boardproductname => D945GCNL
boardserialnumber => BTNL74300AAE
facterversion => 1.6.18
hardwareisa => x86_64
hardwaremodel => x86_64
hostname => dhcppc3
id => root
interfaces => bond0,eth0,eth1,lo,venet0
ipaddress => 192.168.1.33
ipaddress_bond0 => 192.168.1.33
ipaddress_lo => 127.0.0.1
is_virtual => false
kernel => Linux
kernelmajversion => 2.6
kernelrelease => 2.6.32-042stab076.5
kernelversion => 2.6.32
lsbdistcodename => Final
lsbdistdescription => CentOS release 6.4 (Final)
lsbdistid => CentOS
lsbdistrelease => 6.4
lsbmajdistrelease => 6
lsbrelease => :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
macaddress => 00:1C:C0:3E:4B:7E
macaddress_bond0 => 00:1C:C0:3E:4B:7E
macaddress_eth0 => 00:1C:C0:3E:4B:7E
macaddress_eth1 => 00:19:5B:6B:53:22
memoryfree => 2.37 GB
memorysize => 2.80 GB
memorytotal => 2.80 GB
mtu_bond0 => 1500
mtu_eth0 => 1500
mtu_eth1 => 1500
mtu_lo => 16436
mtu_venet0 => 1500
netmask => 255.255.255.0
netmask_bond0 => 255.255.255.0
netmask_lo => 255.0.0.0
network_bond0 => 192.168.1.0
network_lo => 127.0.0.0
operatingsystem => CentOS
operatingsystemrelease => 6.4
osfamily => RedHat
path => /usr/java/jdk1.7.0_09/bin:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
physicalprocessorcount => 1
processor0 => Intel(R) Pentium(R) Dual  CPU  E2160  @ 1.80GHz
processor1 => Intel(R) Pentium(R) Dual  CPU  E2160  @ 1.80GHz
processorcount => 2
ps => ps -ef
rubysitedir => /usr/lib/ruby/site_ruby/1.8
rubyversion => 1.8.7
selinux => false
sshdsakey => AAAAB3NzaC1kc3MAAACBAKryNPR/FSsHtV0BQrt4kS9PEcYEzy/r8IrVcfPR2JjV4xUOLj8FPHlDGa0vaXeqnt6PF6uKq+v7jyvmTEeHPT8y+Zwd0ldBfWxlNWG/vy1m+nwejpeiWcpZLtZ0Pp4ix/xYqvhgVFk/lSD2VaI4oZDLNvzQUNJcIbUD+HXrLRqtAAAAFQDKCw61lmbXRJFJ1fBSsU6dN2CKNwAAAIBskr0Wy6z6axZhB3sHsONtPZB2lLRyr4IlwSLtsJrEPwbbLxMEYzsLP8hk7O9Bw+ms4vUn4Y/a//8TQ4l8opIqO61dlHPg7RD4+Eh/5Dq52Fs2eZtN7qj1JQUC3Gi8NKQ6FJR+wmndRFZfod4l0EFsrAIeBtOJqVj6u8U60XVQFQAAAIEAgRQQhRzQRGdtJjapID5Kyd9Gq8F1g8KeB0jBoGyEckKtSoZI4l+40nKA6AP6PZ2+AhMtuAHvQEF2QLrThEYuv30vPGmlZerFyBddE4tSo9RB/fCf7Lv2RLnnbAsHooiJRN0az55h10Ry0Belr5+21STGHZB4I2JRKOACzPbeMy4=
sshrsakey => AAAAB3NzaC1yc2EAAAABIwAAAQEAteAWPFoF++NYw0jbQ6dwJjpZrVZrgY85V+1hKDFn+Tjcq64ayDHXEbEhbOXo8PfDpk4S4Qbc+ZBsfdv+4kpAArCW0tCod2AFFDq8fLOtIKUmtF5yel1IjUoqoGqhxTW5JupxRznthtUbFI0/fvf96vVPvmL4bNTGB3Xizo9awc8Hxwf5DHZ+yTXHyLa9JFdKHhupQ127WXniooCke+XM036qN2pwMeCvRwZAKFwdrybP+J4g9jsmDeq2NJBOaFUSkg1zMqg0frjIGvyVNx451AtHeQ/6wXnFLASDQsuuoe4GrKenxtYF9v5YmNxQW5UJahe9HnkJC0gepIgoNH2iew==
swapfree => 4.88 GB
swapsize => 4.88 GB
timezone => IST
type => Unknown
uniqueid => a8c02101
uptime => 1:25 hours
uptime_days => 0
uptime_hours => 1
uptime_seconds => 5136
virtual => openvzhn

3) To get the value for a specific fact, say for example, the OS architechture,

# facter architecture
x86_64

history - shell built in command

In Bash shell, the following three variables help in configuring the history command

1. HISTFILE - Where the history  of the exited shell is stored
2. HISTFILESIZE - How many entries are stored there
3. HISTSIZE - How many entries are kept in the memory while using the shell.

How  to check if history is enabled or disabled for a user?

Run the following command
              set -o | grep history

The output of the above command shall be either history off or history  on

Eg:  $ set -o | grep history
       history         on

How to enable history for a user?

If  set -o | grep history provides the output as history off, then history can be enabled for the user as follows

1) Add the follow line in the file ~/.bashrc

        set -o history

2) Then check the values of the parameters - HISTFILE, HISTSIZE, HISTFILESIZE

          echo $HISTFILE
          echo $HISTSIZE
          echo $HISTFILESIZE

If the value of echo $HISTFILE is blank or /dev/null, add the following line in ~/.bashrc file

          HISTFILE=$HOME/.bash_history

If HISTSIZE or HISTFILESIZE is set to 0, set it to a value of choice in ~/.bashrc file

        HISTFILESIZE=1000
        HISTSIZE=1000

Sorting Processes by CPU and Memory usage

To identify processes causing high CPU usage

ps aux --sort -pcpu | head
ps -aeo pcpu,pid,user,args | sort -k1 -r | head -10

To identify processes causing memory usage

ps aux --sort -rss | head
top -b -n 1 -d 3 -m
ps -Ao vsz,args | sort -n

ps -e -o pid,vsz,comm= | sort -n -k 2

To find CPU usage of threads in a process like Java(Tomcat)

ps -mo pid,lwp,stime,time,cpu -C java

MTU, Jumbo Frames, MSS

MTU

Physical and link layers mandate a Maximum Transmission Unit, MTU. For Ethernet, MTU standard size is defined as 1500 bytes - this is the maximum size of payload in a Ethernet Frame, including the encapsulated TCP and IP headers. All machines/routers in the route of the packet must agree on the same MTU size. 

# ip link show | grep mtu
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc pfifo_fast master bond0 state DOWN qlen 1000
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
4: venet0: <BROADCAST,POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
5: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP

If the packet arriving at a router is too big for the MTU of the next hop network, it may do any of the following

1. Router splits the packet into fragments sized for that MTU, or
2. Router uses ICMP to alert senders when fragmentation is needed, and
3. Sender adjusts frame size appropriately.

Path MTU Discovery(pMTU):

One approach to solve the fragmentation problem is to find the smallest MTU size supported by routers on a path in adavance. This helps in avoiding fragmentation by not sending packets larger than the smallest MTU size supported by routers in the path/link. In Path MTU discovery, an initial packet is sent with "Don't Fragment" flag set. Router that would fragment the packet, drops it and responds to the sender with ICMP "fragmentation needed" error packet marked with the MTU of the next hop. The kernel can cache the path MTU(pMTU) information for a connection locally in the routing cache(/proc/net/rt_cache) and resend smaller packets.

Path MTU discovery is enabled on by default and can be verified by checking the sysctl setting - net.ipv4.ip_no_pmtu_disc

# sysctl -a | grep pmtu
net.ipv4.ip_no_pmtu_disc = 0

This can be turned off with syctl by setting net.ipv4.ip_no_pmtu_disc to 1.

A tool for simulating the Path MTU discovery is tracepath.

However, if a router in the path filters all the ICMP traffic, Path MTU discovery(pMTU) gets broken and therefore affects performance by either

1. Causing intermediate fragmentation at routers, or
2. Causing hosts to assume that the minimum path MTU allowed is that of the IP(576 bytes) when higher MTUs are possible.

Note:

1. Filtering of all UDP, ICMP 'time-exceeded', ICMP 'dest-unreach/port-unreach' breaks the tracepath and traceroute.
2. traceroute -I and mtr commands break if ICMP 'time-exceeded' and ICMP 'echo-request'/'echo-reply' are filtered.

Jumbo Frames:

Standard payload size(MTU) is defined as 1500 bytes. To increase performance, payload size(MTU) can be increased. By increasing the packet size,

1. Number of packets transmitted is reduced
2. Less packets means number of interrupts to be handled is also reduced(CPU service time), thus smaller CPU load
3. Less space on an average is needed for headers

Jumbo frames are Ethernet frames with more than 1,500 bytes of payload (MTU). Conventionally, jumbo frames can carry up to 9,000 bytes of payload. Using a larger MTU value (jumbo frames) can significantly speed up network transfers. All the networking hardware in the route of a jumbo frame must support jumbo frames.

Why jumbo frames cannot exceeed 9000 bytes?

Because the CRC field isn’t long enough to guarantee detection of errors for frames larger than 9000 bytes.

How to set MTU size greater than 1500 bytes for a NIC?

1) ip link set ethX mtu Y

               or

     ifconfig ethX mtu Y

where,
ethX is the ethernet adapter (eth0, eth1, etc.) and
Y is the MTU size of the frame(1500, 4000, 9000).

2) Disable path MTU discovery, as follows

net.ipv4.ip_no_pmtu_disc = 1

MSS

MSS is the Maximum Segment Size. This is usually the local MTU minus TCP/IP headers.

For example, if MTU is 1500 bytes, MSS is usually 1460 bytes(1500 - TCP header 20 bytes - IP header 20 bytes).

screen command in Linux

Using screen command, we can create a session and run multiple windows in it.
First we create a screen session and there we create & run multiple windows.

Commands related to session

1) Create a screen session

screen
screen <command> E.g. screen top

2) Naming a screen session while starting a new session

           screen -S <sessionname>

3) Renaming the current screen session

           a) Ctrl + a

           b) Then type
               :<current_sessionname>  <new_SessionName>

4) Renaming without attaching the screen session
   screen -X  <old_sessionname>   <new_sessionname>

5) Detaching from a current screen session

         Ctrl+a d

6) Detaching from a screen session using screen command

        screen -d <sessionname>

7) Reattaching a screen session

        screen -r <sessionname>

Commands related to windows in screen session

1) Create a new window

          Ctrl+a c

2) To navigate between windows in a screen session

            Ctrl+a p  or Ctrl+a n

3) To list all the windows in a screen session

            Ctrl+a "

4) To name current window in a screen session

             Ctrl+a A

5) To kill a window

              Ctrl+a k
 
  A [y/n] option will be prompted. Typing y will kill the window.

  Else, typing exit from the windows shell shall kill the window.

Curl : Accessing a url through a proxy server from command line

To access a URL through a proxy server from the command line, curl comes handy


$ curl -x http://<myproxy.com>:<port_no>   http://google.co.in

Suppose if the proxy server is password protected, try the following

$ curl -U user:password -x http://<myproxy.com>:<port_no>   http://google.co.in

Connecting to a remote website: Explained

From a browser, when we access a URL, let us examine what happens actually.

In the client side, part of the URL is parsed into a hostname and the hostname is translated into a IP address. The browser is assigned an unprivileged port(for example TCP port 14000) for the connection. A HTTP message is constructed for the web server. Its encapsulated in a TCP message, wrapped in an IP packet header and sent out to the web server listening on port 80.

Now a TCP connection need to be established with the server using a three-way handshake :

When the client program(say, browser) sends it's first connection message, the SYN flag is accompanied by a synchronization sequence number. This sequence number is used as the starting point to number all the rest of the data bytes the client will send.

On the server machine, the kernel hands SYN to the server. The server is listening on port 80 and it's notified of an incoming connection request (the SYN connection synchronization request flag) from the source IP address and port socket pair(client IP address,14000). Server calls accept(). The server allocates a new socket on it's end(web server IP address, 80) and associates it with the client socket.

The kernel in the server machine, sends an acknowledgment(ACK) to the SYN message sent by client,along with it's own synchronization request(SYN) - SYN/ACK. The connection is now half open.

Along with the ACK flag, the server includes the client's sequence number incremented by one. The purpose of ACK flag is to acknowledge the data the client referred by it's sequence number. The server acknowledges this by incrementing client's sequence number - sequence number plus one is the next data byte the server expects to receive. So now the client is free to throw away it's original SYN message since the server has acknowledged the receipt of it.

The server also sets the SYN flag in it's first message. Similar to the client's first message, the SYN flag is accompanied by a synchronization sequence number. The server is passing along it's own starting sequence number for it's half of the connection.

The first message is the only message the server will send with the SYN flag set. This and all subsequent messages have the ACK flag set. The presence of the ACK in all server messages, as compared to lack of an ACK flag in the client's first message is a critical difference.

The client machine receives the SYN/ACK message sent by server and replies with it's own acknowledgement(ACK). Now the TCP three-way handshake is complete and the connection is ESTABLISHED.

From here on, both the client and server set the ACK flag. The SYN flag won't be set by either program.

TCP three-way handshake can be briefed as follows:

1) Client end
    Client sends SYN

2) Server end
    Kernel hands SYN to Server
    Server calls accept()
    Kernel sends SYN/ACK to the client

3) Client end
   Client sends ACK

TCP Connection is ESTABLISHED.

Now the first http request data is passed from client to server.
In the server end, the kernel passes connection into the server's accept() method.

So, a  TCP connection for a typical HTTP request looks as follows

SYN (client->server), 
SYN/ACK (server->client), 
ACK (client->server) - TCP Handshake Complete. TCP Connection Established 

Now the request data is passed from client to server

HTTP Request Data (client->server)

Illustration with example

Let us illustrate it with an output of tcpdump command.

Suppose Iam accessing the website 74.125.236.183 (www.google.co.in) using curl command as follows


$ curl -IL 74.125.236.183
HTTP/1.1 200 OK
Date: Sun, 21 Apr 2013 07:48:49 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
Set-Cookie: NID=67=Y-udEdGqbMrB-O-d_Hw09tH2nhhZJ6LmLNt-LAGXVymAq1Fzejl5qFZFdK68DwpE6wxzVLZ7KJFTucQ2zIs6MLxD7KY3MiR2XBqcGvMrmBc5eUaMn-W6Tw7XStpL9QhI; expires=Mon, 21-Oct-2013 07:48:49 GMT; path=/; domain=.; HttpOnly
P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Transfer-Encoding: chunked

Using tcpdump command, the above transaction shall be captured as follows

# tcpdump -w google.pcap -i bond0 host 74.125.236.183

Press Ctrl+C once the above curl command is complete

Examine the tcpdump command output in file google.pcap as follows

Here, 192,168.1.33 is the client IP
          74.125.236.183 is the server IP

# tcpdump -nnr google.pcap

reading from file google.pcap, link-type EN10MB (Ethernet)
13:20:36.724978 IP 192.168.1.33.41002 > 74.125.236.183.80: Flags [S], seq 3247485552, win 14600, options [mss 1460,sackOK,TS val 1544059 ecr 0,nop,wscale 7], length 0
13:20:36.756286 IP 74.125.236.183.80 > 192.168.1.33.41002: Flags [S.], seq 3589360082, ack 3247485553, win 62392, options [mss 1430,sackOK,TS val 977680945 ecr 1544059,nop,wscale 6], length 0
13:20:36.756326 IP 192.168.1.33.41002 > 74.125.236.183.80: Flags [.], ack 1, win 115, options [nop,nop,TS val 1544091 ecr 977680945], length 0

So far, TCP three-way handshake is complete and TCP connection is established between the client and the server.

After the TCP connection is established, now the first request data is passed from client to server

13:20:36.756402 IP 192.168.1.33.41002 > 74.125.236.183.80: Flags [P.], seq 1:171, ack 1, win 115, options [nop,nop,TS val 1544091 ecr 977680945], length 170

13:20:36.788779 IP 74.125.236.183.80 > 192.168.1.33.41002: Flags [.], ack 171, win 992, options [nop,nop,TS val 977680978 ecr 1544091], length 0
13:20:36.844585 IP 74.125.236.183.80 > 192.168.1.33.41002: Flags [P.], seq 1:598, ack 171, win 992, options [nop,nop,TS val 977681032 ecr 1544091], length 597
13:20:36.844603 IP 192.168.1.33.41002 > 74.125.236.183.80: Flags [.], ack 598, win 124, options [nop,nop,TS val 1544179 ecr 977681032], length 0

Data transfer is complete 

13:20:36.844773 IP 192.168.1.33.41002 > 74.125.236.183.80: Flags [F.], seq 171, ack 598, win 124, options [nop,nop,TS val 1544179 ecr 977681032], length 0
13:20:36.875132 IP 74.125.236.183.80 > 192.168.1.33.41002: Flags [F.], seq 598, ack 172, win 992, options [nop,nop,TS val 977681064 ecr 1544179], length 0
13:20:36.875166 IP 192.168.1.33.41002 > 74.125.236.183.80: Flags [.], ack 599, win 124, options [nop,nop,TS val 1544209 ecr 977681064], length 0


A very interesting read

http://igoro.com/archive/what-really-happens-when-you-navigate-to-a-url/

List all installed Python modules/packages

To list all the installed python packages/modules

1) As super user(root), run the following command


# python -c "help('modules')"

Please wait a moment while I gather a list of all available modules...

BaseHTTPServer      chunk               invest              repr
Bastion             cmath               io                  resource
CDROM               cmd                 iotop               rexec
CGIHTTPServer       code                ipaclient           rfc822
CORBA               codecs              ipalib              rlcompleter
ConfigParser        codeop              ipapython           robotparser
Cookie              collections         itertools           rpm
Crypto              colorsys            iwlib               rpmUtils
DLFCN               commands            ixf86config         runpy
DocXMLRPCServer     compileall          json                scanext
HTMLParser          compiler            kerberos            scdate
IN                  contextlib          keyword             sched
MimeWriter          cookielib           krbV                sckdump
ORBit               copy                ldap                scservices
OpenSSL             copy_reg            ldapurl             select
PortableServer      cracklib            ldif                selinux
Queue               createrepo          lib2to3             sets
SSSDConfig          crypt               libiscsi            setuptools
SimpleHTTPServer    cryptsetup          libproxy            sgmllib
SimpleXMLRPCServer  csv                 libuser             sha
SocketServer        ctypes              libxml2             shelve
StringIO            cups                libxml2mod          shlex
TYPES               cupsext             linecache           shutil
UserDict            cupshelpers         linuxaudiodev       signal
UserList            curl                locale              site
UserString          curses              logging             slip
_LWPCookieJar       datetime            lxml                smbc
_MozillaCookieJar   dbhash              macpath             smtpd
__builtin__         dbm                 macurl2path         smtplib
__future__          dbus                mailbox             snack
_abcoll             dbus_bindings       mailcap             sndhdr
_anthy              decimal             mako                socket
_ast                decorator           markupbase          sos
_bisect             default_encoding_utf8 markupsafe          spwd
_bsddb              deltarpm            marshal             sqlite3
_bytesio            difflib             math                sqlitecachec
_codecs             dircache            md5                 sre
_codecs_cn          dis                 meh                 sre_compile
_codecs_hk          distutils           mhlib               sre_constants
_codecs_iso2022     dl                  mimetools           sre_parse
_codecs_jp          dmidecode           mimetypes           ssl
_codecs_kr          dmidecodemod        mimify              stat
_codecs_tw          doctest             mmap                statvfs
_collections        drv_libxml2         modulefinder        string
_cracklib           dsextras            multifile           stringold
_crypt              dsml                multiprocessing     stringprep
_csv                dumbdbm             mutex               strop
_ctypes             dummy_thread        netaddr             struct
_curses             dummy_threading     netrc               subprocess
_curses_panel       easy_install        new                 sunau
_dbus_bindings      egg                 nis                 sunaudio
_dbus_glib_bindings email               nntplib             symbol
_deltarpm           encodings           nose                symtable
_elementtree        errno               nss                 sys
_fileio             ethtool             ntpath              syslog
_functools          exceptions          nturl2path          system_config_keyboard
_gamin              fcntl               numbers             tabnanny
_hashlib            feedparser          numpy               talloc
_heapq              filecmp             opcode              tarfile
_hotshot            fileinput           operator            telnetlib
_json               firstboot           optparse            tempfile
_ldap               fnmatch             orca                termios
_locale             formatter           os                  test
_lsprof             fpformat            os2emxpath          textwrap
_multibytecodec     fractions           ossaudiodev         this
_multiprocessing    ftplib              packagekit          thread
_ped                functools           pango               threading
_random             future_builtins     pangocairo          time
_snack              gamin               paramiko            timeit
_socket             gc                  parser              timing
_sqlite3            gconf               parted              toaiff
_sqlitecache        gdbm                pcardext            token
_sre                genericpath         pdb                 tokenize
_ssl                getopt              pickle              trace
_strptime           getpass             pickletools         traceback
_struct             gettext             pip                 tty
_symtable           gio                 pipes               types
_threading_local    glib                pkg_resources       unicodedata
_warnings           glob                pkgutil             unittest
_weakref            gmenu               platform            uno
abc                 gnome               plistlib            unohelper
abrt_exception_handler gnomeapplet         popen2              urlgrabber
acutil              gnomecanvas         poplib              urllib
aifc                gnomekeyring        posix               urllib2
anthy               gnomevfs            posixfile           urlparse
anydbm              gobject             posixpath           user
array               gpgme               pprint              uu
ast                 grp                 profile             uuid
asynchat            gst                 pstats              vte
asyncore            gstoption           pty                 warnings
atexit              gtk                 pwd                 wave
atk                 gtksourceview2      py_compile          weakref
audiodev            gtkunixprint        pyatspi             webbrowser
audioop             gzip                pyclbr              webkit
base64              hashlib             pycryptsetup        whichdb
bdb                 heapq               pycurl              wireshark_be
beaker              hmac                pydoc               wireshark_gen
binascii            hotshot             pydoc_topics        wnck
binhex              hpmudext            pyexpat             wsgiref
bisect              htmlentitydefs      pygst               xdg
block               htmllib             pygtk               xdrlib
bonobo              httplib             pyhbac              xf86config
bsddb               ibus                pykickstart         xml
bz2                 idlelib             pynotify            xmllib
cPickle             ihooks              pysss               xmlrpclib
cProfile            imageop             pysss_murmur        xxsubtype
cStringIO           imaplib             quopri              yum
cairo               imghdr              random              yumutils
calendar            imp                 re                  zipfile
cas                 imputil             readline            zipimport
cgi                 iniparse            report              zlib
cgitb               inspect             reportclient

Enter any module name to get more help.  Or, type "modules spam" to search for modules whose descriptions contain the word "spam".

2) Alternatively, pip-python command too can be used to list the installed python packages/modules


$ pip-python freeze
Beaker==1.3.1
Mako==0.3.4
MarkupSafe==0.9.2
SSSDConfig==1.9.2
cas==0.15
cups==1.0
cupshelpers==1.0
decorator==3.0.1
distribute==0.6.10
ethtool==0.6
feedparser==5.0.1
firstboot==1.110
freeipa==2.0.0.alpha.0
iniparse==0.3.1
iotop==0.3.2
ipapython==3.0.0
iwlib==1.0
kerberos==1.0
lxml==2.2.3
netaddr==0.7.5
nose==0.10.4
numpy==1.4.1
paramiko==1.7.5
pyOpenSSL==0.10
pycrypto==2.0.1
pycryptsetup==0.0.11
pycurl==7.19.0
pygpgme==0.1
pykickstart==1.74.12
python-default-encoding==0.1
python-dmidecode==3.10.13
python-ldap==2.3.10
python-meh==0.11
python-nss==0.13
pyxdg==0.18
scdate==1.9.60
sckdump==2.0.5
scservices==0.99.45
scservices.dbus==0.99.45
slip==0.2.20
slip.dbus==0.2.20
slip.gtk==0.2.20
smbc==1.0
urlgrabber==3.9.1
yum-metadata-parser==1.1.2

Install python libraries/packages/modules in CentOS

To install python libraries/packages/modules in CentOS, there are two ways to do it

1) Using yum
2) Using pip (Python Package Index) - The python package pip provides a tool named pip-python for installing and managing Python packages

To install a python library/package/module using yum

For this purpose, EPEL(Extra Packages for Enterprise Linux) repo for yum need to be installed first.

1) Find the version of Linux


# lsb_release -a
LSB Version:    :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID: CentOS
Description:    CentOS release 6.4 (Final)
Release:        6.4
Codename:       Final

2) Download and install the epel repo rpm relevant for the above version

a) # wget http://mirror-fpt-telecom.fpt.net/fedora/epel/6/x86_64/epel-release-6-8.noarch.rpm

b) # yum install epel-release-6-8.noarch.rpm

Once the EPEL yum repo is installed. any python package/module/library can be installed using yum like any other linux package

Say, for example, if I want to install feedparser python library/package/module, it can done as follows

# yum install python-feedparser

To install a python library/package/module using pip

1) pip is a python module. So, first python pip module need to be installed. To install the python module, EPEL repo on CentOS need to be enabled(How to enable EPEL repo is mentioned in above step).

2) Once EPEL repo is enabled, install pip as follows

      # yum install python-pip

This provides us a tool called pip-python, using which, the following can be done

a) Install a package

      # pip-python install packagename

b) Uninstall a package

      # pip-python uninstall packagename

c) To list all installed python packages/modules/libraries

      # pip-python freeze

d) To get help with pip-python command

       #  pip-python help

Flags in TCP Header

The 20-byte TCP header has seven inividual bit flags - URG, ACK, PSH, SYN, FIN, RST, Placeholder
While using tcpdump, these flags are represented as urg, ack, p, s, f, r, .

URG - urg - Indicates that the urgent pointer portion of the header should be examined. Urgent data should take precedence over other data. For example, pressing Ctrl-C to terminate a download.

ACK - ack -Indicates that the Acknowledgement number should be examined. Ack packet is used the acknowledge the receipt of data. This flag may appear in conjuction with other flags.

PSH - P -Indicates that the receiver should hand this data upto the next layer as soon as possible. Signals the immediate push of data from sending host to the receiving host.

SYN - S - Initiates a connection. SYN packet, a session establishment request. First part of any TCP connection.

FIN - F - Indicates that the sender(either client or server) is done sending data. It indicates the intention to terminate the existing connection to the other end.

RST - R - Indicates that the connection should be reset. It indicates the sender's intention to immediately abort the existing connection.

Placeholder - . - If the connection does not have a syn, finish, rest or push flag set, this placeholder flag will be found after the destination port. Note that it also appears in conjunction with the ack flag.

A sample tcpdump report, where we are accessing the host 74.125.236.183.80 on it's port 80 using curl command

For the following command

# curl -IL 74.125.236.183

the tcpdump report generated using the command

# tcpdump -w test.pcap -i bond0 host 74.125.236.183

# tcpdump -nnr test.pcap

13:20:36.724978 IP 192.168.1.33.41002 > 74.125.236.183.80: Flags [S], seq 3247485552, win 14600, options [mss 1460,sackOK,TS val 1544059 ecr 0,nop,wscale 7], length 0
13:20:36.756286 IP 74.125.236.183.80 > 192.168.1.33.41002: Flags [S.], seq 3589360082, ack 3247485553, win 62392, options [mss 1430,sackOK,TS val 977680945 ecr 1544059,nop,wscale 6], length 0
13:20:36.756326 IP 192.168.1.33.41002 > 74.125.236.183.80: Flags [.], ack 1, win 115, options [nop,nop,TS val 1544091 ecr 977680945], length 0
13:20:36.756402 IP 192.168.1.33.41002 > 74.125.236.183.80: Flags [P.], seq 1:171, ack 1, win 115, options [nop,nop,TS val 1544091 ecr 977680945], length 170
13:20:36.788779 IP 74.125.236.183.80 > 192.168.1.33.41002: Flags [.], ack 171, win 992, options [nop,nop,TS val 977680978 ecr 1544091], length 0
13:20:36.844585 IP 74.125.236.183.80 > 192.168.1.33.41002: Flags [P.], seq 1:598, ack 171, win 992, options [nop,nop,TS val 977681032 ecr 1544091], length 597
13:20:36.844603 IP 192.168.1.33.41002 > 74.125.236.183.80: Flags [.], ack 598, win 124, options [nop,nop,TS val 1544179 ecr 977681032], length 0
13:20:36.844773 IP 192.168.1.33.41002 > 74.125.236.183.80: Flags [F.], seq 171, ack 598, win 124, options [nop,nop,TS val 1544179 ecr 977681032], length 0
13:20:36.875132 IP 74.125.236.183.80 > 192.168.1.33.41002: Flags [F.], seq 598, ack 172, win 992, options [nop,nop,TS val 977681064 ecr 1544179], length 0
13:20:36.875166 IP 192.168.1.33.41002 > 74.125.236.183.80: Flags [.], ack 599, win 124, options [nop,nop,TS val 1544209 ecr 977681064], length 0

How a TCP Connection is closed?

Closing a TCP connection is a four step process as opposed to three way process for connection establishment. The extra step is due to the full duplex nature of the TCP connection where both client and server may be sending data at any given time.

1. TCP connection closing is initiated either by client or server, by sending a TCP segment with FIN flag set to the other end indicating that it is wishing to close the connection. So if client sends a TCP segment with FIN flag set to the server, then server is said to be on the CLOSE_WAIT state and client is said to be on  FIN_WAIT_1 state.
2. After the FIN is received by the server, the server sends a TCP segment with ACK flag set to the client, incrementing the sequence number by one. Now the client goes into FIN_WAIT_2 state. The server also indicates to it's own higher layer protocols that the connection is terminated.
3. The server closes the connection by sending a TCP segemnt with FIN flag set to the client. This causes the server to go into LAST_ACK state while the client goes into TIME_WAIT state.
4. Finally, the client acknowledges the FIN sent by the server with an ACK and increments the sequence number by one. This causes the connection to go into CLOSED state.

Because TCP connections can be closed by either side, a TCP connection can exist in half-closed mode in which one end has initiated the FIN sequence but the other end has not done so.

RST - TCP connections can also be terminated with one end sending a TCP segment with RST(reset) flag set. This informs the other side to use an abortive release method. This is opposed to the normal termination of TCP connection sometimes referred to as orderly release.

Why TCP is called connection oriented protocol? TCP three-way handshake

Before exchanging data, a connection must be established between client and server using a three-way handshake. Hence TCP is called a connection oriented protocol.

1. The client while initiating a connection with the server, sends a TCP segment with SYN flag set, as well as an Initial Sequence Number (ISN) and the port number of the server. This client is said to be in SYN_SENT state.
2.  At this point, the server will make an entry for connection in the listen queue  and send back a reply that has both SYN and ACK flags set which indicates that the server acknowledges receiving the client's initial packet and wishes to establish a connection with the client. The server responds with a TCP segment with SYN flag and ACK flag set. Also, the server sets the ISN with a value one higher than the ISN sent by the client. This is referred as SYN-ACK packet or SYN-ACK segment. The server is said to be in SYN_RCVD state.
3. The client then acknowledges the SYN-ACK packet be sending a TCP segment with ACK flag set and by incrementing ISN by one. This completes the three-way handshake and connection is said to be in ESTABLISHED state. Once the client responds, the connection  moves from listen queue into the connection queue.

For a system on a high-latency network that receives a large number of connection requests, there is a possibility of listen queue becoming full because of the time required for the clients to complete the connection.

With TCP, both client and server can send data at the same time, making TCP a full duplex protocol.

TCP three-way handshake can be briefed as follows:

1) Client end
    Client sends SYN

2) Server end
    Kernel hands SYN to Server
    Server calls accept()
    Kernel sends SYN/ACK to the client

3) Client end
   Client sends ACK

TCP Connection is ESTABLISHED.

Now in the server end, the kernel passes connection into the server's accept() menthod.