Linux performance observability tools

Openstack 5 on CentOS 7

CentOS 7 Install minimal.

Step 1 wget http://dl.fedoraproject.org/pub/epel/beta/7/x86_64/epel-release-7-0.2.noarch.rpm
Step 2 vi /etc/sysconfig/selinux  # Set:  SELINUX=permissive
Step 3 yum update -y
Step 4 systemctl disable NetworkManager
Step 5 yum install ntp -y
Step 6 systemctl disable firewalld #Openstack dont understand firewalld rules
Step 7 vi /etc/ntp.conf
server 1.in.pool.ntp.org
server 1.asia.pool.ntp.org
server 0.asia.pool.ntp.org
Step 8 vi /etc/hosts
192.168.122.113 guggle.co.in. # Openstack need static ip address.
Step 9 yum install -y http://rdo.fedorapeople.org/rdo-release.rpm
Step 10 yum install -y openstack-packstack
Step 11 packstack --gen-answer-file=/root/answers.txt
Step 12 packstack --answer-file=/root/answers.txt
Welcome to Installer setup utility
Packstack changed given value  to required value /root/.ssh/id_rsa.pub

Installing:
Clean Up                                             [ DONE ]
Setting up ssh keys                                  [ DONE ]
Discovering hosts' details                           [ DONE ]
Adding pre install manifest entries                  [ DONE ]
Preparing servers                                    [ DONE ]
Adding AMQP manifest entries                         [ DONE ]
Adding MySQL manifest entries                        [ DONE ]
Adding Keystone manifest entries                     [ DONE ]
Adding Glance Keystone manifest entries              [ DONE ]
Adding Glance manifest entries                       [ DONE ]
Adding Cinder Keystone manifest entries              [ DONE ]
Adding Cinder manifest entries                       [ DONE ]
Checking if the Cinder server has a cinder-volumes vg[ DONE ]
Adding Nova API manifest entries                     [ DONE ]
Adding Nova Keystone manifest entries                [ DONE ]
Adding Nova Cert manifest entries                    [ DONE ]
Adding Nova Conductor manifest entries               [ DONE ]
Creating ssh keys for Nova migration                 [ DONE ]
Gathering ssh host keys for Nova migration           [ DONE ]
Adding Nova Compute manifest entries                 [ DONE ]
Adding Nova Scheduler manifest entries               [ DONE ]
Adding Nova VNC Proxy manifest entries               [ DONE ]
Adding Openstack Network-related Nova manifest entries[ DONE ]
Adding Nova Common manifest entries                  [ DONE ]
Adding Neutron API manifest entries                  [ DONE ]
Adding Neutron Keystone manifest entries             [ DONE ]
Adding Neutron L3 manifest entries                   [ DONE ]
Adding Neutron L2 Agent manifest entries             [ DONE ]
Adding Neutron DHCP Agent manifest entries           [ DONE ]
Adding Neutron LBaaS Agent manifest entries          [ DONE ]
Adding Neutron Metering Agent manifest entries       [ DONE ]
Adding Neutron Metadata Agent manifest entries       [ DONE ]
Adding OpenStack Client manifest entries             [ DONE ]
Adding Horizon manifest entries                      [ DONE ]
Adding Swift Keystone manifest entries               [ DONE ]
Adding Swift builder manifest entries                [ DONE ]
Adding Swift proxy manifest entries                  [ DONE ]
Adding Swift storage manifest entries                [ DONE ]
Adding Swift common manifest entries                 [ DONE ]
Adding Provisioning Demo manifest entries            [ DONE ]
Adding MongoDB manifest entries                      [ DONE ]
Adding Ceilometer manifest entries                   [ DONE ]
Adding Ceilometer Keystone manifest entries          [ DONE ]
Adding Nagios server manifest entries                [ DONE ]
Adding Nagios host manifest entries                  [ DONE ]
Adding post install manifest entries                 [ DONE ]
Installing Dependencies                              [ DONE ]
Copying Puppet modules and manifests                 [ DONE ]
Applying 192.168.122.113_prescript.pp
192.168.122.113_prescript.pp:                        [ DONE ]          
Applying 192.168.122.113_amqp.pp
Applying 192.168.122.113_mysql.pp
192.168.122.113_amqp.pp:                             [ DONE ]      
192.168.122.113_mysql.pp:                            [ DONE ]      
Applying 192.168.122.113_keystone.pp
Applying 192.168.122.113_glance.pp
Applying 192.168.122.113_cinder.pp
192.168.122.113_keystone.pp:                         [ DONE ]        
192.168.122.113_cinder.pp:                           [ DONE ]        
192.168.122.113_glance.pp:                           [ DONE ]        
Applying 192.168.122.113_api_nova.pp
192.168.122.113_api_nova.pp:                         [ DONE ]        
Applying 192.168.122.113_nova.pp
192.168.122.113_nova.pp:                             [ DONE ]    
Applying 192.168.122.113_neutron.pp
192.168.122.113_neutron.pp:                          [ DONE ]        
Applying 192.168.122.113_neutron_fwaas.pp
Applying 192.168.122.113_osclient.pp
Applying 192.168.122.113_horizon.pp
192.168.122.113_neutron_fwaas.pp:                    [ DONE ]              
192.168.122.113_osclient.pp:                         [ DONE ]              
192.168.122.113_horizon.pp:                          [ DONE ]              
Applying 192.168.122.113_ring_swift.pp
192.168.122.113_ring_swift.pp:                       [ DONE ]          
Applying 192.168.122.113_swift.pp
Applying 192.168.122.113_provision_demo.pp
192.168.122.113_swift.pp:                            [ DONE ]              
192.168.122.113_provision_demo.pp:                   [ DONE ]              
Applying 192.168.122.113_mongodb.pp
192.168.122.113_mongodb.pp:                          [ DONE ]        
Applying 192.168.122.113_ceilometer.pp
Applying 192.168.122.113_nagios.pp
Applying 192.168.122.113_nagios_nrpe.pp
192.168.122.113_ceilometer.pp:                       [ DONE ]            
192.168.122.113_nagios_nrpe.pp:                      [ DONE ]            
192.168.122.113_nagios.pp:                           [ DONE ]            
Applying 192.168.122.113_postscript.pp
192.168.122.113_postscript.pp:                       [ DONE ]          
Applying Puppet manifests                            [ DONE ]
Finalizing                                           [ DONE ]

 **** Installation completed successfully ******


Additional information:
 * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components.
 * Did not create a cinder volume group, one already existed
 * File /root/keystonerc_admin has been created on OpenStack client host 192.168.122.113. To use the command line tools you need to source the file.
 * To access the OpenStack Dashboard browse to http://192.168.122.113/dashboard .
Please, find your login credentials stored in the keystonerc_admin in your home directory.
 * To use Nagios, browse to http://192.168.122.113/nagios username: nagiosadmin, password: 55209fe74e694f3c
 * The installation log file is available at: /var/tmp/packstack/20140808-001328-M74eSH/openstack-setup.log
 * The generated manifests are available at: /var/tmp/packstack/20140808-001328-M74eSH/manifests

RHEL 7 /CentOS 7 Boot Single User Mode

1. init=/bin/sh at the end of line that starts with linux in Grub2 (In case of  VMWare like KVM or VirtualBox use rb.break instead of inti=/bin/sh)
2. run mount -o remount,rw /sysroot when the system boots
3. run chroot /sysroot
4. run passwd
5. run touch /.autorelabel
6. run exit to leave chroot
7. run exit to logout

RHEL 7 Minimal Installation

Using "weechat" config Gtalk.

Installation weechat on Fedora 20

yum -y install weechat libnotify notify-python wget python-xmpp

Configuration

$weechat-curses

Setup scripts

$ cd ~/.weechat/python/autoload
$ wget http://weechat.org/files/scripts/weeget.py  

/python load weeget
/weeget
/weechat install buffers
/set weechat.bar.buffers.position top
/weechat install jabber

/jabber add gtalk rajatjpatel@gmail.com yousetthatpasswdisworkingnow talk.google.com:5223
/jabber connect gtalk

To install Docker on Fedora

Docker-based container sandbox provides a number of advantages for application deployment environment, such as lightweight isolation, deployment portability, ease of maintenance, etc.

Why docker?
• Smaller than VMs • Improved performance • Secure • Flexible

Not only for the cloud environment, Docker can also be quite useful for end users, especially when you want to test out particular software under a specific Linux environment. You can easily spin up a Docker container for the target environment, install and test the software in it, and then throw away the container once you are done. The whole process from beginning to end is quite efficient, and you can avoid messing up your end system all along.

In this post, I am going to describe how to create and manage Docker containers on Fedora.

To install Docker on Fedora, use the following commands:

# yum install docker-io
# systemctl start docker.service
# systemctl enable docker.service

Basic Usage of Docker
To start a new Docker container, you need to decide which Docker image to use for the container. You can search the official Docker image index which lists publicly available Docker images. The Docker index includes Linux base images managed by Docker team (e.g., Ubuntu, Debian, Fedora, CentOS), as well as user-contributed custom images (e.g., MySQL, Redis, WordPress).

For example, to start a Fedora/Ubuntu container in the interactive mode, run the following command. The last argument '/bin/bash' is to be executed inside a container upon its launch.

docker run -i -t ubuntu /bin/bash or docker pull ubuntu /docker pull fedora

The first time you run the above command, it will download available Ubuntu docker image(s) over networks, and then boot up a Docker container using the image. A Ubuntu container will boot up instantly, and you will see a console prompt inside the container. You can access a full-fledged Ubuntu operating system inside the container sandbox.

list of all containers

docker ps -a

Start container of your choice

docker start [container-id]

Remove container from you local repo

docker rm [container-id]

Running container in order to view or interact with the container

docker attach [container-id]

To remove a container image from the local repository:

docker rmi [image-id]

To search a container image from repositry

docker search fedora or docker search centos

Linux Performance Analysis and Tuning

What is “tuned” ?

Tuning profile delivery mechanism

Red Hat ships tuned profiles that improve performance for many workloads...hopefully yours!

To install tuned:

# yum install tuned -y

Now start the services provided by tuned:

# service tuned start

# chkconfig tuned on

# service ktune start

# chkconfig ktune on

To find the current active profile and state of service:

# tuned-adm active
Current active profile: default
Service tuned: enabled, running
Service ktune: enabled, running

To list all the available profiles:

# tuned-adm list
Available profiles:
- default
- throughput-performance
- laptop-ac-powersave
- spindown-disk
- desktop-powersave
- laptop-battery-powersave
- latency-performance
- server-powersave
- enterprise-storage
Current active profile: default

To switch to a different profile:
# tuned-adm profile spindown-disk
NOTE: spindown-disk is one of the profiles

Each profile has 4 configuration file under /etc/tune-profiles/<profile-name>. If you want to create a profile of your own, simply copy one of the profile directory with a different name, change the config files inside it according to your own requirement and activate it.
# cd /etc/tune-profiles/
# cp -a default myprofile
# cd myprofile
# ls
ktune.sh  ktune.sysconfig  sysctl.ktune  tuned.conf

# tuned-adm list
Available profiles:
- balanced
- desktop
- latency-performance
- powersave
- sap
- throughput-performance
- virtual-guest
- virtual-host
Current active profile: balanced

# tuned-adm profile myprofile

In case if you want to disable all tuning, then run:
# tuned-adm off or #server tuned stop

# tuned-adm profile throughput-performance
# tuned-adm active
Current active profile: throughput-performance
# time taskset -c 0 seq 1 60000000 > /dev/null

real 0m0.689s <--
user 0m0.676s
sys 0m0.012s
# service tuned stop
Redirecting to /bin/systemctl stop  tuned.service
# time taskset -c 0 seq 1 60000000 > /dev/null

real 0m0.698s <--
user 0m0.686s
sys 0m0.012s

Above sample from laptop.

# uname -a

Linux rajat.patel.fc20 3.14.2-200.fc20.x86_64 #1 SMP Mon Apr 28 14:40:57 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Guidelines on hardening Linux server installation

A. Use the least amount of permissions to accomplish the required task.
B. Use the minimum of software tools and packages to implement the required services.
C. Securing your server is a continuous process, not a one-time activity.

1. Always start with a minimal server installation and add packages as they're needed. Reasoning: every piece of software can be a potential vulnerability. There's no need to insert a vulnerability with something you'll not use in the first place.

2. Set up a user account and install sudo. Add user to the sudoers file and configure system to allow root login only on tty1-tty8.

3. Install ssh and reconfigure it to listen on non-standard efemeral port (> 1024). If possible, install port knocking to unlock the new ssh port - do not use default knocking sequence!

4. Configure ssh to disable password authentication and permit only pubkey authentication. Install your public key in authorized_keys of a user account. Always use strong passphrase for your keys and keep private keys as best as you can!

5. If server requires ordinary users to log in onto it, configure PAM to harden password policy, if possible. If a users that logs in do not require full access to command, give him a restricted shell.

6. Install only necessary services for your server. If you can choose, choose services that implement some kind of encryption when accessing server. For example, if your users need some sort of remote file services and they can use both FTP and SCP/SFTP, choose SCP/SFTP. Avoid telnet service if at all possible (but have telnet client as you'll probably need it when troubleshooting tcp connections).

7. Use SELinux or AppArmor if you can. Learn to create custom SELinux policies if needed (some software just won't work with SELinux in enforcing mode).

8. Set up iptables in the most restrictive way. On INPUT chain block all ports except those that your services use on that server. Limit open ports by IP addresses that are permitted to access them, if at all possible.

9. Set rules to the OUTPUT chain as well. Lots of exploits work by establishing connection from compromised server back to the attacker's machine which usually bypass external firewalls. Limiting outgoing traffic can mitigate attacks and render them useless.

10. Implement remote central log server and install some sort of log analyzing software. Check logs frequently and search for unusual patterns.

11. Check your /etc/fstab and add 'nodev,noexec,nosuid' options on filesystems that will not have executables and devices. This is far from bullet-proof protection and it can be thwarted by competent attacker, but can still stop some script kiddies and automated attacks.

12. Use chroot when possible. I know this is also almost trivial to evade, but still, why would I ease the attacker's job?

13. Implement tripwire or similar software if you can keep your file-signature database on some non-volatile media (like CD-ROM).

14. Upgrade and apply patches if at all possible.

15. Run some audit tool, both local (Lynis) and remote (OpenVAS, Nessus) to check if you managed to cover all the bases. Analyze reports made by those apps and apply necessary changes to your system.

List of Network Diagnostic Tools

In RHEL/Centos/Fedora
ip -- show / manipulate routing, devices, policy routing and tunnels
ifconfig --configure a network interface
ethtool -- query or control network driver and hardware settings
tcpdump -- dump traffic on a network
wireshark --  Interactively dump and analyze network traffic
netstat -- Print network connections, routing tables, interface statistics, masquerade connections, and multicast memberships
ss -- another utility to investigate sockets
dropwatch -- dropwatch aims to consolidate several of those checks into one tool, making it easier for a sysadmin or developer to detect lost packets.
systemtap --  systemtap script translator/driver
nmap -- Network exploration tool and security / port scanner
nc -- arbitrary TCP and UDP connections and listens
ping -- send ICMP ECHO_REQUEST to network hosts
ping6 -- send ICMP ECHO_REQUEST to network hosts
iptables -- administration tool for IPv4 packet filtering and NAT
ip6tables -- administration tool for IPv6 packet filtering and NAT
arp -- Linux ARP kernel module.
arping -- send ARP REQUEST to a neighbour host
tc -- show / manipulate traffic control settings, is used to configure Traffic Control in the Linux
lnstat -- unified linux network statistics
nstat -- rtacct - network statistics tools
traceroute -- print the route packets trace to network host
tracepath -- traces path to a network host discovering MTU along this path
tunctl --create and manage persistent TUN/TAP interfaces