Monday, August 26, 2013

Get Rid of Deleted Open Files

Sharing a nice articles which I found on web. Hope it will be useful to you as well..

You might have this scenario; Logfiles deleted while the process is still running. That's annoying: On your Linux-Server the /var filesystem is nearly full. You remove a very large logfile that you don't need with the rm command:

rajat@root-centos## df -Ph /var
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/root-var  7.1G  7.0G  100M  99% /var
rajat@root-centos## ls -l /var/log/myapp/userlog
rajat@root-centos## rm /var/log/myapp/userlog
rajat@root-centos## df -Ph /var
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/root-var  7.1G  7.0G  100M  99% /var

But what's that? The filesystem is still full. With lsof you can see, that the logfile is still opened in write mode:

rajat@root-centos## lsof | grep var/log/myapp/userlog
myapp    25139      root   4w     REG      3,12       0    2101404 /var/log/myapp/userlog (deleted)

To actually free up the space you would have to stop the logging process. But since this might be a mission critical application this is not always an option. Is there any way to get rid of the file without stopping the logging process?

Find the deleted file's representation under /proc

Actually we cannot remove the file as long as the file is still in use by a process. But what we can do is: Getting the size down to 0. Thanks to Linux' enhanced /proc filesystem.

And that's how you do it:

First find the process that still uses the file (we already did that - see above):

rajat@root-centos## lsof | grep var/log/myapp/userlog
myapp    25139      root   4w     REG      3,12       0    2101404 /var/log/myapp/userlog (deleted)
lsof tells us that a process with PID=25139 has opened the file (with number 4) in write mode. See the bolded part of the lsof output.

Knowing the PID of the process and the file number we can visit its representation under /proc:

rajat@root-centos## cd /proc/25139/fd
rajat@root-centos#:/proc/25139/fd#  ls -l 4
lr-x------ 1 root root 64 2010-01-07 17:10 4 -> /var/log/myapp/userlog (deleted)
We can do almost everything with this file (called 4 here) what we can do with a real file: we can less it, copy it, and we can change its contents!

Free up the Space

As already said, we cannot remove the file, but what we can do is getting the size down to zero. And that's done as with every other file, e.g. if you use bash (or ksh) - what is most likely under Linux:

rajat@root-centos## > /proc/25139/fd/4
rajat@root-centos## df -Ph /var
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/root-var  7.1G  4.9G  1.2G  69% /var
As we see there is again free space under /var, and the process is still running:

rajat@root-centos## lsof | grep var/log/myapp/userlog
myapp    25139      root   4w     REG      3,12       0        0  /var/log/myapp/userlog (deleted)

What more could be done...?

As said before you can work on this file as you work on real files. That means, you could even save this file and compress it before getting its size down to 0:

rajat@root-centos## cp /proc/25139/fd/4 /tmp/userlog
rajat@root-centos## gzip -9 /tmp/userlog
rajat@root-centos## mv /tmp/userlog.gz /var/log/myapp/userlog.1.gz
rajat@root-centos## > /proc/25139/fd/4

Final Remarks

This procedure helps you if you are in a pinch - but basically you should never remove such an open file, because you still have an issue here: The process still writes to this file - and the only way to see what it logs is to use the procedure to save the file as shown above.

So remember to bring the size down to 0 in the first place instead:

rajat@root-centos## > /var/log/myapp/userlog
This way the space in the filesystem is freed immediately - and you still see what your application is writing to this file.

Monday, June 10, 2013

Cent OS 6.4 as PuppetMaster 3.2.1 installation and configuration

What is Puppet?

Puppet is IT automation software that helps system administrators manage infrastructure throughout its

lifecycle, from provisioning and configuration to patch management and compliance. Using Puppet, you

can easily automate repetitive tasks, quickly deploy critical applications, and proactively manage

change, scaling from 10s of servers to 1000s, on-premise or in the cloud.

How Puppet Works

Puppet uses a declarative, model-based approach to IT automation.

Define the desired state of the infrastructure’s configuration using Puppet’s declarative

configuration language.
Simulate configuration changes before enforcing them.
Enforce the deployed desired state automatically, correcting any configuration drift.
Report on the differences between actual and desired states and any changes made enforcing the desired


Puppet Master Server Install & Configuration

#vi /etc/hosts master client

#yum install puppet-server -y

#vi /etc/puppet/puppet.conf
dns_alt_name =

#vi /etc/sysconfig/puppet

#service puppetmaster restart && chkconfig puppetmaster on

#puppet cert list
#puppet cert sign

Now we need create "Puppet Policy" @/etc/puppet/manifest/site.pp

[root@master ~]# cat /etc/puppet/manifests/site.pp
# Create "/tmp/testfile" if it doesn't exist.
class test_class {
    file { "/tmp/testfile":
       ensure => present,
       mode   => 600,
       owner  => root,
       group  => root

# tell puppet on which client to run the class
node default {
    include test_class

file {'/etc/puppet/files/MobileWorld.jar':
  ensure => present,
  mode   => '0777',
  owner  => 'root',
  group  => 'root',
  source => 'puppet:///files/MobileWorld.jar',
file {'/etc/motd':
                        source => 'puppet:///files/motd',

file {'/etc/puppet/files/mMobile.jar':
  ensure => present,
  mode   => '0777',
  owner  => 'root',
  group  => 'root',
  source => 'puppet:///files/mMobile.jar',

[root@master ~]#

Puppet Slave Install & Configuration

#vi /etc/hosts
#vi /etc/hosts client master

#yum install puppet -y

#vi /etc/puppet/puppet.conf
server =

#vi /etc/default/puppet

#service puppet restart && chkconfig puppet on

#puppet agent --test

At the client end need write a one crontab.

* * * 1 * puppet --onetime --no-daemonize --logdest syslog > /dev/null 2>&1

Wednesday, May 8, 2013

Rescan LUN's without rebooting RedHat 6 /Cent OS

# ls /sys/class/fc_host
 host0  host1  host2  host3

#echo "c t l" >  /sys/class/scsi_host/hostH/scan

where H is the HBA number, c is the channel on the HBA, t is the SCSI target ID, and l is the LUN.

Thursday, March 14, 2013

ISOLATE Spacewalk for RHEL 6.1

The following is a example to have a RHEL6.1 clone channel:
#spacewalk-create-channel -l -s -v 6 -s Server -u U1 -a x86_64 –d rhel6-1-channel –N “RHEL 6.1”
-l <username>
The Satellite username (admin or personal account), this account should have enough authorization to create / clone channels.
-s <server>
The Satellite server that needs the new channel, if no channel is listed localhost will be used.
-v <version>
The version of the channel to create (e.g. 5, 4, 3, 2.1).
-s <release>
The release of the channel to create (e.g. AS, ES,  WS,  Server, Client, Desktop).
-u <Update level>
The update level of the channel to create (e.g. GOLD, U1, U2, U3, U4, U5, U6, U7, U8, U9), where GOLD stands for the initial release.
-a <Architecture>
The  arch  of the channel to create (e.g. i386, ia64, ppc, s390, s390x, x86_64).
-d <label>
The label of the destination channel. This will be created if not present.
-N <title>
If the destination channel is created use DEST_NAME for it's name. If not provided the label will be used.