1. Identify the Group name of MySQL Packages
yum grouplist displays all package groups that are available in the repository. As shown below, mysql package group is called “MySQL Database”.
# yum grouplist | grep -i mysql
MySQL Database
2. What is bundled in the “MySQL Database” group?
yum groupinfo displays all the packages that are bundled in a group. This displays the mandatory, default and optional packages that are available in that particular group.
As shown below, “MySQL Database” group contains 1 mandatory package, 6 default packages, and 5 optional packages.
# yum groupinfo "MySQL Database"
Group: MySQL Database
Description: This package group contains packages useful for use with MySQL.
Mandatory Packages:
mysql
Default Packages:
MySQL-python
libdbi-dbd-mysql
mysql-connector-odbc
mysql-server
perl-DBD-MySQL
unixODBC
Optional Packages:
mod_auth_mysql
mysql-bench
mysql-devel
php-mysql
qt-MySQL
3. Install the “MySQL Database” group using yum groupinstall
yum groupinstall will install the “MySQL Database” group of packages as shown below.
# yum groupinstall "MySQL Database"
Resolving Dependencies
Dependencies Resolved
Transaction Summary
=========================
Install 12 Package(s)
Update 0 Package(s)
Remove 0 Package(s)
Installed:
MySQL-python.i386 0:1.2.1-1 libdbi-dbd-mysql.i386 0:0.8.1a-1.2.2
mysql.i386 0:5.0.77-4.el5_4.2 mysql-connector-odbc.i386 0:3.51.26r1127-1.el5
mysql-server.i386 0:5.0.77-4.el5_4.2 perl-DBD-MySQL.i386 0:3.0007-2.el5
unixODBC.i386 0:2.2.11-7.1
Dependency Installed:
libdbi.i386 0:0.8.1-2.1 libdbi-drivers.i386 0:0.8.1a-1.2.2
libtool-ltdl.i386 0:1.5.22-7.el5_4
mx.i386 0:2.0.6-2.2.2 perl-DBI.i386 0:1.52-2.el5
Complete!
Note: If you are having some issues during the installation, verify the full mysql install log to see what you are missing.
4. Verify MySQL Installation
Execute rpm -qa, to confirm that the mysql related packages are installed.
# rpm -qa | grep -i mysql
MySQL-python-1.2.1-1
mysql-5.0.77-4.el5_4.2
mysql-connector-odbc-3.51.26r1127-1.el5
mysql-server-5.0.77-4.el5_4.2
libdbi-dbd-mysql-0.8.1a-1.2.2
perl-DBD-MySQL-3.0007-2.el5
Check the /etc/passwd and /etc/group to make sure it has created a mysql username and group.
# grep mysql /etc/passwd
mysql:x:27:27:MySQL Server:/var/lib/mysql:/bin/bash
# grep mysql /etc/group
mysql:x:27:
5. MySQL Post installation – Execute mysql_install_db
mysql_install_db program will setup the necessary grant tables. The mysql_install_db program gets executed as part of the rpm installation. But, it doesn’t hurt to execute the mysql_install_db program again to make sure the grant tables are setup properly.
# /usr/bin/mysql_install_db --user=mysql
Installing MySQL system tables...OK
Filling help tables...OK
.....
The latest information about MySQL is available on the web at http://www.mysql.com
6. Start MySQL Server
# service mysqld status
mysqld is stopped
# service mysqld start
Starting MySQL: [ OK ]
7. Verify that the MySQL server is up and running.
# /usr/bin/mysqladmin version
/usr/bin/mysqladmin Ver 8.41 Distrib 5.0.77, for redhat-linux-gnu on i686
Copyright (C) 2000-2006 MySQL AB
This software comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to modify and redistribute it under the GPL license
Server version 5.0.77
Protocol version 10
Connection Localhost via UNIX socket
UNIX socket /var/lib/mysql/mysql.sock
Uptime: 39 sec
Threads: 1 Questions: 2 Slow queries: 0 Opens: 12 Flush tables: 1
Open tables: 6 Queries per second avg: 0.051
# /usr/bin/mysqlshow
+--------------------+
| Databases |
+--------------------+
| information_schema |
| mysql |
| test |
+--------------------+
# /usr/bin/mysqlshow mysql
Database: mysql
+---------------------------+
| Tables |
+---------------------------+
| columns_priv |
| db |
| func |
| help_category |
| time_zone_transition |
| time_zone_transition_type |
| user |
+---------------------------+
Stop and start the mysql server again to make sure they are no issues.
# service mysqld stop
Stopping MySQL: [ OK ]
# service mysqld start
Starting MySQL: [ OK ]
8. Change the MySQL root account password
Change the MySQL root account password to something secure.
# mysql -u root
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.0.77 Source distribution
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> select host, user from mysql.user;
+-----------+------+
| host | user |
+-----------+------+
| 127.0.0.1 | root |
| localhost | |
| localhost | root |
+-----------+------+
5 rows in set (0.00 sec)
mysql> set password for 'root'@'localhost' = PASSWORD('DoNotTell$AnyBody');
Query OK, 0 rows affected (0.00 sec)
mysql> set password for 'root'@'127.0.0.1' = PASSWORD('DoNotTell$AnyBody');
Query OK, 0 rows affected (0.00 sec)
Make sure you are able to login to MySQL using the new password as shown below.
# mysql -u root
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)
# mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 7
Server version: 5.0.77 Source distribution
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql>
Thursday, April 29, 2010
Wednesday, April 28, 2010
File System use for Linux
Ext3
Ext3 is a journaling filesystem developed by Stephen Tweedie. It is compatible to ext2 filesystems; actually you can look at it as an ext2 filesystem with a journal file. The journaling capability means no more waiting for fsck’s or worrying about metadata corruption. What is most noticeable is that you can switch back and forth between ext2 and ext3 on a partition without any problem: it is just a matter of giving the mount command the right filesystem type.
Ext3 FAQ
JFS
The Journaled File System (JFS) provides a log-based, byte-level file system that was developed for transaction-oriented, high performance systems. Scalable and robust, its advantage over non-journaled file systems is its quick restart capability: JFS can restore a file system to a consistent state in a matter of seconds or minutes.
While tailored primarily for the high throughput and reliability requirements of servers ( from single processor systems to advanced multi-processor and clustered systems), JFS is also applicable to client configurations where performance and reliability are desired.
JFS is IBM software.
JFS Overview at IBM Developer Works
ZFS
ZFS is a new kind of file system that provides simple administration, transactional semantics, end-to-end data integrity, and immense scalability.
ZFS includes a pooled storage model, is always consistent on disk, offers protection from data corruption, does live data scrubbing, makes instantaneous snapshots and clones, has fast native backup and restore capabilities, is highly scalable, has built in compression and a simplified administration model.
ZFS is shipped with OpenSolaris.
OpenSolaris Community: ZFS
XFS
XFS is a high-performance journaling filesystem. XFS combines advanced journaling technology with full 64-bit addressing and scalable structures and algorithms. This combination delivers the most scalable high-performance filesystem ever conceived.
XFS is SGI software.
XFS homepage at SGI Developer Central
VXFS
VXFS is the Veritas Filesystem and is distributed through Symantec, who bought Veritas quite a while ago.
Symantec Homepage
Remote
NFS
The Network Filesystem is available on every UNIX-like OS.
Wikipedia about NFS
Open AFS
AFS is a distributed filesystem product, pioneered at Carnegie Mellon University and supported and developed as a product by Transarc Corporation (now IBM Pittsburgh Labs). It offers a client-server architecture for file sharing, providing location independence, scalability, security, and transparent migration capabilities for data.
IBM branched the source of the AFS product, and made a copy of the source available for community development and maintenance. They called the release OpenAFS.
OpenAFS Homepage
CIFS/SAMBA
Samba is an Open Source implementation of Microsoft’s Common Internet Filesystem (CIFS).
CIFS itself is part of all Microsoft Windows versions.
Samba support for IRIX can be bought from SGI itself.
Samba Homepage
CIFS or Public SMB Information on Common Internet File System
Samba for IRIX FAQ
Distributed
Lustre
Lustre is a scalable, secure, robust, highly-available cluster file system. It is designed, developed and maintained by Cluster File Systems, Inc.
The central goal is the development of a next-generation cluster file system which can serve clusters with 10,000′s of nodes, petabytes of storage, move 100′s of GB/sec with state of the art security and management infrastructure.
Lustre runs today on many of the largest Linux clusters in the world, and is included by CFS’s partners as a core component of their cluster offering (examples include HP StorageWorks SFS, and the Cray XT3 and XD1 supercomputers). Today’s users have also demonstrated that Lustre scales down as well as it scales up, and run in production on clusters as small as 4 and as large as 15,000 nodes.
The latest version of Lustre is always available from Cluster File Systems, Inc. Public Open Source releases of Lustre are made under the GNU General Public License. These releases are found here, and are used in production supercomputing environments worldwide.
You may subscribe to the lustre-announce mailing list to be informed of releases.
Lustre development would not have been possible without funding and guidance from many organizations, including several US National Laboratories, early adopters, and product partners.
Lustre Homepage
GPFS
The IBM General Parallel File System (GPFS) is a high-performance shared-disk file system that can provide fast, reliable data access from all nodes in a homogenous or heterogenous cluster of IBM UNIX® servers running either the AIX 5L or the Linux operating system.
GPFS allows parallel applications simultaneous access to a set of files (or even a single file) from any node that has the GPFS file system mounted while providing a high level of control over all file system operations.
GPFS is IBM software.
GPFS Homepage at IBM
GFS
GFS (Global File System) is a cluster file system. It allows a cluster of computers to simultaneously use a block device that is shared between them (with FC, iSCSI, NBD, etc…). GFS reads and writes to the block device like a local filesystem, but also uses a lock module to allow the computers coordinate their I/O so filesystem consistency is maintained. One of the nifty features of GFS is perfect consistency — changes made to the filesystem on one machine show up immediately on all other machines in the cluster.
GFS Homepage at Red Hat
PVFS
The goal of the Parallel Virtual File System (PVFS) Project is to explore the design, implementation, and uses of parallel I/O. PVFS serves as both a platform for parallel I/O research as well as a production file system for the cluster computing community. PVFS is currently targeted at clusters of workstations, or Beowulfs.
PVFS supports the UNIX I/O interface and allows existing UNIX I/O programs to use PVFS files without recompiling. The familiar UNIX file tools (ls, cp, rm, etc.) will all operate on PVFS files and directories as well. This is accomplished via a Linux kernel module which is provided as a separate package.
PVFS stripes file data across multiple disks in different nodes in a cluster. By spreading out file data in this manner, larger files can be created, potential bandwidth is increased, and network bottlenecks are minimized. A 64-bit interface is implemented as well, allowing large (more than 2GB) files to be created and accessed.
The Parallel Virtual Filesystem Project
PVFS 2
Ext3 is a journaling filesystem developed by Stephen Tweedie. It is compatible to ext2 filesystems; actually you can look at it as an ext2 filesystem with a journal file. The journaling capability means no more waiting for fsck’s or worrying about metadata corruption. What is most noticeable is that you can switch back and forth between ext2 and ext3 on a partition without any problem: it is just a matter of giving the mount command the right filesystem type.
Ext3 FAQ
JFS
The Journaled File System (JFS) provides a log-based, byte-level file system that was developed for transaction-oriented, high performance systems. Scalable and robust, its advantage over non-journaled file systems is its quick restart capability: JFS can restore a file system to a consistent state in a matter of seconds or minutes.
While tailored primarily for the high throughput and reliability requirements of servers ( from single processor systems to advanced multi-processor and clustered systems), JFS is also applicable to client configurations where performance and reliability are desired.
JFS is IBM software.
JFS Overview at IBM Developer Works
ZFS
ZFS is a new kind of file system that provides simple administration, transactional semantics, end-to-end data integrity, and immense scalability.
ZFS includes a pooled storage model, is always consistent on disk, offers protection from data corruption, does live data scrubbing, makes instantaneous snapshots and clones, has fast native backup and restore capabilities, is highly scalable, has built in compression and a simplified administration model.
ZFS is shipped with OpenSolaris.
OpenSolaris Community: ZFS
XFS
XFS is a high-performance journaling filesystem. XFS combines advanced journaling technology with full 64-bit addressing and scalable structures and algorithms. This combination delivers the most scalable high-performance filesystem ever conceived.
XFS is SGI software.
XFS homepage at SGI Developer Central
VXFS
VXFS is the Veritas Filesystem and is distributed through Symantec, who bought Veritas quite a while ago.
Symantec Homepage
Remote
NFS
The Network Filesystem is available on every UNIX-like OS.
Wikipedia about NFS
Open AFS
AFS is a distributed filesystem product, pioneered at Carnegie Mellon University and supported and developed as a product by Transarc Corporation (now IBM Pittsburgh Labs). It offers a client-server architecture for file sharing, providing location independence, scalability, security, and transparent migration capabilities for data.
IBM branched the source of the AFS product, and made a copy of the source available for community development and maintenance. They called the release OpenAFS.
OpenAFS Homepage
CIFS/SAMBA
Samba is an Open Source implementation of Microsoft’s Common Internet Filesystem (CIFS).
CIFS itself is part of all Microsoft Windows versions.
Samba support for IRIX can be bought from SGI itself.
Samba Homepage
CIFS or Public SMB Information on Common Internet File System
Samba for IRIX FAQ
Distributed
Lustre
Lustre is a scalable, secure, robust, highly-available cluster file system. It is designed, developed and maintained by Cluster File Systems, Inc.
The central goal is the development of a next-generation cluster file system which can serve clusters with 10,000′s of nodes, petabytes of storage, move 100′s of GB/sec with state of the art security and management infrastructure.
Lustre runs today on many of the largest Linux clusters in the world, and is included by CFS’s partners as a core component of their cluster offering (examples include HP StorageWorks SFS, and the Cray XT3 and XD1 supercomputers). Today’s users have also demonstrated that Lustre scales down as well as it scales up, and run in production on clusters as small as 4 and as large as 15,000 nodes.
The latest version of Lustre is always available from Cluster File Systems, Inc. Public Open Source releases of Lustre are made under the GNU General Public License. These releases are found here, and are used in production supercomputing environments worldwide.
You may subscribe to the lustre-announce mailing list to be informed of releases.
Lustre development would not have been possible without funding and guidance from many organizations, including several US National Laboratories, early adopters, and product partners.
Lustre Homepage
GPFS
The IBM General Parallel File System (GPFS) is a high-performance shared-disk file system that can provide fast, reliable data access from all nodes in a homogenous or heterogenous cluster of IBM UNIX® servers running either the AIX 5L or the Linux operating system.
GPFS allows parallel applications simultaneous access to a set of files (or even a single file) from any node that has the GPFS file system mounted while providing a high level of control over all file system operations.
GPFS is IBM software.
GPFS Homepage at IBM
GFS
GFS (Global File System) is a cluster file system. It allows a cluster of computers to simultaneously use a block device that is shared between them (with FC, iSCSI, NBD, etc…). GFS reads and writes to the block device like a local filesystem, but also uses a lock module to allow the computers coordinate their I/O so filesystem consistency is maintained. One of the nifty features of GFS is perfect consistency — changes made to the filesystem on one machine show up immediately on all other machines in the cluster.
GFS Homepage at Red Hat
PVFS
The goal of the Parallel Virtual File System (PVFS) Project is to explore the design, implementation, and uses of parallel I/O. PVFS serves as both a platform for parallel I/O research as well as a production file system for the cluster computing community. PVFS is currently targeted at clusters of workstations, or Beowulfs.
PVFS supports the UNIX I/O interface and allows existing UNIX I/O programs to use PVFS files without recompiling. The familiar UNIX file tools (ls, cp, rm, etc.) will all operate on PVFS files and directories as well. This is accomplished via a Linux kernel module which is provided as a separate package.
PVFS stripes file data across multiple disks in different nodes in a cluster. By spreading out file data in this manner, larger files can be created, potential bandwidth is increased, and network bottlenecks are minimized. A 64-bit interface is implemented as well, allowing large (more than 2GB) files to be created and accessed.
The Parallel Virtual Filesystem Project
PVFS 2
Monday, April 26, 2010
How to *.tar/ *.tar.gz/ *.tgz/ *.tar.bz2
1. Creating an archive using tar command
Creating an uncompressed tar archive using option cvf
This is the basic command to create a tar archive.
$ tar cvf archive_name.tar dirname/
In the above command:
c – create a new archive
v – verbosely list files which are processed.
f – following is the archive file name
Creating a tar gzipped archive using option cvzf
The above tar cvf option, does not provide any compression. To use a gzip compression on the tar archive, use the z option as shown below.
$ tar cvzf archive_name.tar.gz dirname/
z – filter the archive through gzip
Note: .tgz is same as .tar.gz
Creating a bzipped tar archive using option cvjf
Create a bzip2 tar archive as shown below:
$ tar cvjf archive_name.tar.bz2 dirname/
j – filter the archive through bzip2
gzip vs bzip2: bzip2 takes more time to compress and decompress than gzip. bzip2 archival size is less than gzip.
Note: .tbz and .tb2 is same as .tar.bz2
Note: I like to keep the ‘cvf’ option unchanged for all archive creation, and add additional option at the end, which is easier to remember. i.e cvf for archive creation, cvzf for compressed gzip archive creation, cvjf for compressed bzip2 archive creation etc.,
2. Extracting (untar) an archive using tar command
Extract a *.tar file using option xvf
Extract a tar file using option x as shown below:
$ tar xvf archive_name.tar
x – extract files from archive
Extract a gzipped tar archive ( *.tar.gz ) using option xvzf
Use the option z for uncompressing a gzip tar archive.
$ tar xvzf archive_name.tar.gz
Extracting a bzipped tar archive ( *.tar.bz2 ) using option xvjf
Use the option j for uncompressing a bzip2 tar archive.
$ tar xvjf archive_name.tar.bz2
Note: In all the above commands v is optional, which lists the file being processed.
3. Listing an archive using tar command
View the tar archive file content without extracting using option tvf
You can view the *.tar file content before extracting as shown below.
$ tar tvf archive_name.tar
View the *.tar.gz file content without extracting using option tvzf
You can view the *.tar.gz file content before extracting as shown below.
$ tar tvzf archive_name.tar.gz
View the *.tar.bz2 file content without extracting using option tvjf
You can view the *.tar.bz2 file content before extracting as shown below.
$ tar tvjf archive_name.tar.bz2
4. Extract a single file from tar, tar.gz, tar.bz2 file
To extract a specific file from a tar archive, specify the file name at the end of the tar xvf command as shown below. The following command extracts only a specific file from a large tar file.
$ tar -xvf archive_file.tar /path/to/file
Use the relevant option z or j according to the compression method gzip or bzip2 respectively as shown below.
$ tar -xvzf archive_file.tar.gz /path/to/file
$ tar -xvjf archive_file.tar.bz2 /path/to/file
5. Extract a single directory from tar, tar.gz, tar.bz2 file
To extract a single directory (along with it’s subdirectory and files) from a tar archive, specify the directory name at the end of the tar xvf command as shown below. The following extracts only a specific directory from a large tar file.
$ tar -xvf archive_file.tar /path/to/dir/
To extract multiple directories from a tar archive, specify those individual directory names at the end of the tar xvf command as shown below.
$ tar -xvf archive_file.tar /path/to/dir1/ /path/to/dir2/
Use the relevant option z or j according to the compression method gzip or bzip2 respectively as shown below.
$ tar -xvzf archive_file.tar.gz /path/to/dir/
$ tar -xvjf archive_file.tar.bz2 /path/to/dir/
6. Extract group of files from tar, tar.gz, tar.bz2 archives using regular expression
You can specify a regex, to extract files matching a specified pattern. For example, following tar command extracts all the files with pl extension.
$ tar -xvf archive_file.tar --wildcards '*.pl'
Options explanation:
–wildcards *.pl – files with pl extension
7. Adding a file or directory to an existing archive using option -r
You can add additional files to an existing tar archive as shown below. For example, to append a file to *.tar file do the following:
$ tar rvf archive_name.tar newfile
This newfile will be added to the existing archive_name.tar. Adding a directory to the tar is also similar,
$ tar rvf archive_name.tar newdir/
Note: You cannot add file or directory to a compressed archive. If you try to do so, you will get “tar: Cannot update compressed archives” error as shown below.
$ tar -rvzf archive_name.tgz newfile
tar: Cannot update compressed archives
Try `tar --help' or `tar --usage' for more information.
8. Verify files available in tar using option -W
As part of creating a tar file, you can verify the archive file that got created using the option W as shown below.
$ tar cvfW file_name.tar dir/
If you are planning to remove a directory/file from an archive file or from the file system, you might want to verify the archive file before doing it as shown below.
$ tar tvfW file_name.tar
Verify 1/file1
1/file1: Mod time differs
1/file1: Size differs
Verify 1/file2
Verify 1/file3
If an output line starts with Verify, and there is no differs line then the file/directory is Ok. If not, you should investigate the issue.
Note: for a compressed archive file ( *.tar.gz, *.tar.bz2 ) you cannot do the verification.
Finding the difference between an archive and file system can be done even for a compressed archive. It also shows the same output as above excluding the lines with Verify.
Finding the difference between gzip archive file and file system
$ tar dfz file_name.tgz
Finding the difference between bzip2 archive file and file system
$ tar dfj file_name.tar.bz2
9. Estimate the tar archive size
The following command, estimates the tar file size ( in KB ) before you create the tar file.
$ tar -cf - /directory/to/archive/ | wc -c
20480
The following command, estimates the compressed tar file size ( in KB ) before you create the tar.gz, tar.bz2 files.
$ tar -czf - /directory/to/archive/ | wc -c
508
$ tar -cjf - /directory/to/archive/ | wc -c
428
Creating an uncompressed tar archive using option cvf
This is the basic command to create a tar archive.
$ tar cvf archive_name.tar dirname/
In the above command:
c – create a new archive
v – verbosely list files which are processed.
f – following is the archive file name
Creating a tar gzipped archive using option cvzf
The above tar cvf option, does not provide any compression. To use a gzip compression on the tar archive, use the z option as shown below.
$ tar cvzf archive_name.tar.gz dirname/
z – filter the archive through gzip
Note: .tgz is same as .tar.gz
Creating a bzipped tar archive using option cvjf
Create a bzip2 tar archive as shown below:
$ tar cvjf archive_name.tar.bz2 dirname/
j – filter the archive through bzip2
gzip vs bzip2: bzip2 takes more time to compress and decompress than gzip. bzip2 archival size is less than gzip.
Note: .tbz and .tb2 is same as .tar.bz2
Note: I like to keep the ‘cvf’ option unchanged for all archive creation, and add additional option at the end, which is easier to remember. i.e cvf for archive creation, cvzf for compressed gzip archive creation, cvjf for compressed bzip2 archive creation etc.,
2. Extracting (untar) an archive using tar command
Extract a *.tar file using option xvf
Extract a tar file using option x as shown below:
$ tar xvf archive_name.tar
x – extract files from archive
Extract a gzipped tar archive ( *.tar.gz ) using option xvzf
Use the option z for uncompressing a gzip tar archive.
$ tar xvzf archive_name.tar.gz
Extracting a bzipped tar archive ( *.tar.bz2 ) using option xvjf
Use the option j for uncompressing a bzip2 tar archive.
$ tar xvjf archive_name.tar.bz2
Note: In all the above commands v is optional, which lists the file being processed.
3. Listing an archive using tar command
View the tar archive file content without extracting using option tvf
You can view the *.tar file content before extracting as shown below.
$ tar tvf archive_name.tar
View the *.tar.gz file content without extracting using option tvzf
You can view the *.tar.gz file content before extracting as shown below.
$ tar tvzf archive_name.tar.gz
View the *.tar.bz2 file content without extracting using option tvjf
You can view the *.tar.bz2 file content before extracting as shown below.
$ tar tvjf archive_name.tar.bz2
4. Extract a single file from tar, tar.gz, tar.bz2 file
To extract a specific file from a tar archive, specify the file name at the end of the tar xvf command as shown below. The following command extracts only a specific file from a large tar file.
$ tar -xvf archive_file.tar /path/to/file
Use the relevant option z or j according to the compression method gzip or bzip2 respectively as shown below.
$ tar -xvzf archive_file.tar.gz /path/to/file
$ tar -xvjf archive_file.tar.bz2 /path/to/file
5. Extract a single directory from tar, tar.gz, tar.bz2 file
To extract a single directory (along with it’s subdirectory and files) from a tar archive, specify the directory name at the end of the tar xvf command as shown below. The following extracts only a specific directory from a large tar file.
$ tar -xvf archive_file.tar /path/to/dir/
To extract multiple directories from a tar archive, specify those individual directory names at the end of the tar xvf command as shown below.
$ tar -xvf archive_file.tar /path/to/dir1/ /path/to/dir2/
Use the relevant option z or j according to the compression method gzip or bzip2 respectively as shown below.
$ tar -xvzf archive_file.tar.gz /path/to/dir/
$ tar -xvjf archive_file.tar.bz2 /path/to/dir/
6. Extract group of files from tar, tar.gz, tar.bz2 archives using regular expression
You can specify a regex, to extract files matching a specified pattern. For example, following tar command extracts all the files with pl extension.
$ tar -xvf archive_file.tar --wildcards '*.pl'
Options explanation:
–wildcards *.pl – files with pl extension
7. Adding a file or directory to an existing archive using option -r
You can add additional files to an existing tar archive as shown below. For example, to append a file to *.tar file do the following:
$ tar rvf archive_name.tar newfile
This newfile will be added to the existing archive_name.tar. Adding a directory to the tar is also similar,
$ tar rvf archive_name.tar newdir/
Note: You cannot add file or directory to a compressed archive. If you try to do so, you will get “tar: Cannot update compressed archives” error as shown below.
$ tar -rvzf archive_name.tgz newfile
tar: Cannot update compressed archives
Try `tar --help' or `tar --usage' for more information.
8. Verify files available in tar using option -W
As part of creating a tar file, you can verify the archive file that got created using the option W as shown below.
$ tar cvfW file_name.tar dir/
If you are planning to remove a directory/file from an archive file or from the file system, you might want to verify the archive file before doing it as shown below.
$ tar tvfW file_name.tar
Verify 1/file1
1/file1: Mod time differs
1/file1: Size differs
Verify 1/file2
Verify 1/file3
If an output line starts with Verify, and there is no differs line then the file/directory is Ok. If not, you should investigate the issue.
Note: for a compressed archive file ( *.tar.gz, *.tar.bz2 ) you cannot do the verification.
Finding the difference between an archive and file system can be done even for a compressed archive. It also shows the same output as above excluding the lines with Verify.
Finding the difference between gzip archive file and file system
$ tar dfz file_name.tgz
Finding the difference between bzip2 archive file and file system
$ tar dfj file_name.tar.bz2
9. Estimate the tar archive size
The following command, estimates the tar file size ( in KB ) before you create the tar file.
$ tar -cf - /directory/to/archive/ | wc -c
20480
The following command, estimates the compressed tar file size ( in KB ) before you create the tar.gz, tar.bz2 files.
$ tar -czf - /directory/to/archive/ | wc -c
508
$ tar -cjf - /directory/to/archive/ | wc -c
428
Saturday, April 17, 2010
Netstat Command Fedora/Centos/RedHat/
1. List All Ports (both listening and non listening ports)
List all ports using netstat -a
# netstat -a | more
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 localhost:30037 *:* LISTEN
udp 0 0 *:bootpc *:*
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 6135 /tmp/.X11-unix/X0
unix 2 [ ACC ] STREAM LISTENING 5140 /var/run/acpid.socket
List all tcp ports using netstat -at
# netstat -at
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 localhost:30037 *:* LISTEN
tcp 0 0 localhost:ipp *:* LISTEN
tcp 0 0 *:smtp *:* LISTEN
tcp6 0 0 localhost:ipp [::]:* LISTEN
List all udp ports using netstat -au
# netstat -au
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
udp 0 0 *:bootpc *:*
udp 0 0 *:49119 *:*
udp 0 0 *:mdns *:*
2. List Sockets which are in Listening State
List only listening ports using netstat -l
# netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 localhost:ipp *:* LISTEN
tcp6 0 0 localhost:ipp [::]:* LISTEN
udp 0 0 *:49119 *:*
List only listening TCP Ports using netstat -lt
# netstat -lt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 localhost:30037 *:* LISTEN
tcp 0 0 *:smtp *:* LISTEN
tcp6 0 0 localhost:ipp [::]:* LISTEN
List only listening UDP Ports using netstat -lu
# netstat -lu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
udp 0 0 *:49119 *:*
udp 0 0 *:mdns *:*
List only the listening UNIX Ports using netstat -lx
# netstat -lx
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 6294 private/maildrop
unix 2 [ ACC ] STREAM LISTENING 6203 public/cleanup
unix 2 [ ACC ] STREAM LISTENING 6302 private/ifmail
unix 2 [ ACC ] STREAM LISTENING 6306 private/bsmtp
3. Show the statistics for each protocol
Show statistics for all ports using netstat -s
# netstat -s
Ip:
11150 total packets received
1 with invalid addresses
0 forwarded
0 incoming packets discarded
11149 incoming packets delivered
11635 requests sent out
Icmp:
0 ICMP messages received
0 input ICMP message failed.
Tcp:
582 active connections openings
2 failed connection attempts
25 connection resets received
Udp:
1183 packets received
4 packets to unknown port received.
.....
Show statistics for TCP (or) UDP ports using netstat -st (or) -su
# netstat -st
# netstat -su
4. Display PID and program names in netstat output using netstat -p
netstat -p option can be combined with any other netstat option. This will add the “PID/Program Name” to the netstat output. This is very useful while debugging to identify which program is running on a particular port.
# netstat -pt
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 1 0 ramesh-laptop.loc:47212 192.168.185.75:www CLOSE_WAIT 2109/firefox
tcp 0 0 ramesh-laptop.loc:52750 lax:www ESTABLISHED 2109/firefox
5. Don’t resolve host, port and user name in netstat output
When you don’t want the name of the host, port or user to be displayed, use netstat -n option. This will display in numbers, instead of resolving the host name, port name, user name.
This also speeds up the output, as netstat is not performing any look-up.
# netstat -an
If you don’t want only any one of those three items ( ports, or hosts, or users ) to be resolved, use following commands.
# netsat -a --numeric-ports
# netsat -a --numeric-hosts
# netsat -a --numeric-users
6. Print netstat information continuously
netstat will print information continuously every few seconds.
# netstat -c
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 ramesh-laptop.loc:36130 101-101-181-225.ama:www ESTABLISHED
tcp 1 1 ramesh-laptop.loc:52564 101.11.169.230:www CLOSING
tcp 0 0 ramesh-laptop.loc:43758 server-101-101-43-2:www ESTABLISHED
tcp 1 1 ramesh-laptop.loc:42367 101.101.34.101:www CLOSING
^C
7. Find the non supportive Address families in your system
netstat --verbose
At the end, you will have something like this.
netstat: no support for `AF IPX' on this system.
netstat: no support for `AF AX25' on this system.
netstat: no support for `AF X25' on this system.
netstat: no support for `AF NETROM' on this system.
8. Display the kernel routing information using netstat -r
# netstat -r
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
192.168.1.0 * 255.255.255.0 U 0 0 0 eth2
link-local * 255.255.0.0 U 0 0 0 eth2
default 192.168.1.1 0.0.0.0 UG 0 0 0 eth2
Note: Use netstat -rn to display routes in numeric format without resolving for host-names.
9. Find out on which port a program is running
# netstat -ap | grep ssh
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 1 0 dev-db:ssh 101.174.100.22:39213 CLOSE_WAIT -
tcp 1 0 dev-db:ssh 101.174.100.22:57643 CLOSE_WAIT -
Find out which process is using a particular port:
# netstat -an | grep ':80'
10. Show the list of network interfaces
# netstat -i
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0 1500 0 0 0 0 0 0 0 0 0 BMU
eth2 1500 0 26196 0 0 0 26883 6 0 0 BMRU
lo 16436 0 4 0 0 0 4 0 0 0 LRU
Display extended information on the interfaces (similar to ifconfig) using netstat -ie:
# netstat -ie
Kernel Interface table
eth0 Link encap:Ethernet HWaddr 00:10:40:11:11:11
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Memory:f6ae0000-f6b00000
List all ports using netstat -a
# netstat -a | more
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 localhost:30037 *:* LISTEN
udp 0 0 *:bootpc *:*
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 6135 /tmp/.X11-unix/X0
unix 2 [ ACC ] STREAM LISTENING 5140 /var/run/acpid.socket
List all tcp ports using netstat -at
# netstat -at
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 localhost:30037 *:* LISTEN
tcp 0 0 localhost:ipp *:* LISTEN
tcp 0 0 *:smtp *:* LISTEN
tcp6 0 0 localhost:ipp [::]:* LISTEN
List all udp ports using netstat -au
# netstat -au
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
udp 0 0 *:bootpc *:*
udp 0 0 *:49119 *:*
udp 0 0 *:mdns *:*
2. List Sockets which are in Listening State
List only listening ports using netstat -l
# netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 localhost:ipp *:* LISTEN
tcp6 0 0 localhost:ipp [::]:* LISTEN
udp 0 0 *:49119 *:*
List only listening TCP Ports using netstat -lt
# netstat -lt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 localhost:30037 *:* LISTEN
tcp 0 0 *:smtp *:* LISTEN
tcp6 0 0 localhost:ipp [::]:* LISTEN
List only listening UDP Ports using netstat -lu
# netstat -lu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
udp 0 0 *:49119 *:*
udp 0 0 *:mdns *:*
List only the listening UNIX Ports using netstat -lx
# netstat -lx
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 6294 private/maildrop
unix 2 [ ACC ] STREAM LISTENING 6203 public/cleanup
unix 2 [ ACC ] STREAM LISTENING 6302 private/ifmail
unix 2 [ ACC ] STREAM LISTENING 6306 private/bsmtp
3. Show the statistics for each protocol
Show statistics for all ports using netstat -s
# netstat -s
Ip:
11150 total packets received
1 with invalid addresses
0 forwarded
0 incoming packets discarded
11149 incoming packets delivered
11635 requests sent out
Icmp:
0 ICMP messages received
0 input ICMP message failed.
Tcp:
582 active connections openings
2 failed connection attempts
25 connection resets received
Udp:
1183 packets received
4 packets to unknown port received.
.....
Show statistics for TCP (or) UDP ports using netstat -st (or) -su
# netstat -st
# netstat -su
4. Display PID and program names in netstat output using netstat -p
netstat -p option can be combined with any other netstat option. This will add the “PID/Program Name” to the netstat output. This is very useful while debugging to identify which program is running on a particular port.
# netstat -pt
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 1 0 ramesh-laptop.loc:47212 192.168.185.75:www CLOSE_WAIT 2109/firefox
tcp 0 0 ramesh-laptop.loc:52750 lax:www ESTABLISHED 2109/firefox
5. Don’t resolve host, port and user name in netstat output
When you don’t want the name of the host, port or user to be displayed, use netstat -n option. This will display in numbers, instead of resolving the host name, port name, user name.
This also speeds up the output, as netstat is not performing any look-up.
# netstat -an
If you don’t want only any one of those three items ( ports, or hosts, or users ) to be resolved, use following commands.
# netsat -a --numeric-ports
# netsat -a --numeric-hosts
# netsat -a --numeric-users
6. Print netstat information continuously
netstat will print information continuously every few seconds.
# netstat -c
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 ramesh-laptop.loc:36130 101-101-181-225.ama:www ESTABLISHED
tcp 1 1 ramesh-laptop.loc:52564 101.11.169.230:www CLOSING
tcp 0 0 ramesh-laptop.loc:43758 server-101-101-43-2:www ESTABLISHED
tcp 1 1 ramesh-laptop.loc:42367 101.101.34.101:www CLOSING
^C
7. Find the non supportive Address families in your system
netstat --verbose
At the end, you will have something like this.
netstat: no support for `AF IPX' on this system.
netstat: no support for `AF AX25' on this system.
netstat: no support for `AF X25' on this system.
netstat: no support for `AF NETROM' on this system.
8. Display the kernel routing information using netstat -r
# netstat -r
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
192.168.1.0 * 255.255.255.0 U 0 0 0 eth2
link-local * 255.255.0.0 U 0 0 0 eth2
default 192.168.1.1 0.0.0.0 UG 0 0 0 eth2
Note: Use netstat -rn to display routes in numeric format without resolving for host-names.
9. Find out on which port a program is running
# netstat -ap | grep ssh
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 1 0 dev-db:ssh 101.174.100.22:39213 CLOSE_WAIT -
tcp 1 0 dev-db:ssh 101.174.100.22:57643 CLOSE_WAIT -
Find out which process is using a particular port:
# netstat -an | grep ':80'
10. Show the list of network interfaces
# netstat -i
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0 1500 0 0 0 0 0 0 0 0 0 BMU
eth2 1500 0 26196 0 0 0 26883 6 0 0 BMRU
lo 16436 0 4 0 0 0 4 0 0 0 LRU
Display extended information on the interfaces (similar to ifconfig) using netstat -ie:
# netstat -ie
Kernel Interface table
eth0 Link encap:Ethernet HWaddr 00:10:40:11:11:11
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Memory:f6ae0000-f6b00000
Thursday, April 8, 2010
Using bridged networking with Virt-manager
To do this, we install the package bridge-utils...
yum install bridge-utils
... and configure a bridge. Create the file /etc/sysconfig/network-scripts/ifcfg-br0 (please use the BOOTPROTO, BROADCAST, IPADDR, NETMASK and NETWORK values from the /etc/sysconfig/network-scripts/ifcfg-eth0 file):
vi /etc/sysconfig/network-scripts/ifcfg-br0
DEVICE=br0
TYPE=Bridge
BOOTPROTO=static
BROADCAST=192.168.0.255
IPADDR=192.168.0.100
NETMASK=255.255.255.0
NETWORK=192.168.0.0
ONBOOT=yes
Modify /etc/sysconfig/network-scripts/ifcfg-eth0 as follows (comment out BOOTPROTO, BROADCAST, IPADDR, NETMASK, and NETWORK and add BRIDGE=br0):
vi /etc/sysconfig/network-scripts/ifcfg-eth0
# Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+
DEVICE=eth0
#BOOTPROTO=static
#BROADCAST=192.168.0.255
HWADDR=00:10:A7:05:AF:EB
#IPADDR=192.168.0.100
#NETMASK=255.255.255.0
#NETWORK=192.168.0.0
ONBOOT=yes
BRIDGE=br0
Restart the network...
/etc/init.d/network restart
... and run
ifconfig
It should now show the network bridge (br0):
[root@rajat1 ~]# ifconfig
br0 Link encap:Ethernet HWaddr 00:10:A7:05:AF:EB
inet addr:192.168.0.100 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::210:a7ff:fe05:afeb/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:17 errors:0 dropped:0 overruns:0 frame:0
TX packets:53 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1160 (1.1 KiB) TX bytes:14875 (14.5 KiB)
eth0 Link encap:Ethernet HWaddr 00:10:A7:05:AF:EB
inet6 addr: fe80::210:a7ff:fe05:afeb/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:13662 errors:7 dropped:160 overruns:4 frame:0
TX packets:11646 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:15144608 (14.4 MiB) TX bytes:1379942 (1.3 MiB)
Interrupt:74 Base address:0xcc00
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:38 errors:0 dropped:0 overruns:0 frame:0
TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4308 (4.2 KiB) TX bytes:4308 (4.2 KiB)
virbr0 Link encap:Ethernet HWaddr 00:00:00:00:00:00
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:35 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:9987 (9.7 KiB)
[root@rajat1 ~]# brctl show
bridge name bridge id STP enabled interfaces
pan0 8000.000000000000 no
virbr0 8000.000000000000 yes
br0 8000.0019b97ec863 yes eth0
yum install bridge-utils
... and configure a bridge. Create the file /etc/sysconfig/network-scripts/ifcfg-br0 (please use the BOOTPROTO, BROADCAST, IPADDR, NETMASK and NETWORK values from the /etc/sysconfig/network-scripts/ifcfg-eth0 file):
vi /etc/sysconfig/network-scripts/ifcfg-br0
DEVICE=br0
TYPE=Bridge
BOOTPROTO=static
BROADCAST=192.168.0.255
IPADDR=192.168.0.100
NETMASK=255.255.255.0
NETWORK=192.168.0.0
ONBOOT=yes
Modify /etc/sysconfig/network-scripts/ifcfg-eth0 as follows (comment out BOOTPROTO, BROADCAST, IPADDR, NETMASK, and NETWORK and add BRIDGE=br0):
vi /etc/sysconfig/network-scripts/ifcfg-eth0
# Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+
DEVICE=eth0
#BOOTPROTO=static
#BROADCAST=192.168.0.255
HWADDR=00:10:A7:05:AF:EB
#IPADDR=192.168.0.100
#NETMASK=255.255.255.0
#NETWORK=192.168.0.0
ONBOOT=yes
BRIDGE=br0
Restart the network...
/etc/init.d/network restart
... and run
ifconfig
It should now show the network bridge (br0):
[root@rajat1 ~]# ifconfig
br0 Link encap:Ethernet HWaddr 00:10:A7:05:AF:EB
inet addr:192.168.0.100 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::210:a7ff:fe05:afeb/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:17 errors:0 dropped:0 overruns:0 frame:0
TX packets:53 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1160 (1.1 KiB) TX bytes:14875 (14.5 KiB)
eth0 Link encap:Ethernet HWaddr 00:10:A7:05:AF:EB
inet6 addr: fe80::210:a7ff:fe05:afeb/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:13662 errors:7 dropped:160 overruns:4 frame:0
TX packets:11646 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:15144608 (14.4 MiB) TX bytes:1379942 (1.3 MiB)
Interrupt:74 Base address:0xcc00
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:38 errors:0 dropped:0 overruns:0 frame:0
TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4308 (4.2 KiB) TX bytes:4308 (4.2 KiB)
virbr0 Link encap:Ethernet HWaddr 00:00:00:00:00:00
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:35 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:9987 (9.7 KiB)
[root@rajat1 ~]# brctl show
bridge name bridge id STP enabled interfaces
pan0 8000.000000000000 no
virbr0 8000.000000000000 yes
br0 8000.0019b97ec863 yes eth0
Subscribe to:
Posts (Atom)