Monday, August 16, 2010

How to Recover Deleted Files in Fedora / Ubuntu

foremost is a forensics application to recover files based on their headers, footers, and internal data structures. Foremost can work on image files, such as those generated by dd, Safeback, Encase, etc, or directly on a drive. This short article shows how you can use foremost to recover deleted files.

Currently foremost can recover the following file types:
  • jpg - Support for the JFIF and Exif formats including implementations used in modern digital cameras.
  • gif
  • png
  • bmp - Support for windows bmp format.
  • avi
  • exe - Support for Windows PE binaries, will extract DLL and EXE files along with their compile times.
  • mpg - Support for most MPEG files (must begin with 0x000001BA)
  • wav
  • riff - This will extract AVI and RIFF since they use the same file format (RIFF). note faster than running each separately.
  • wmv - Note may also extract -wma files as they have similar format.
  • mov
  • pdf
  • ole - This will grab any file using the OLE file structure. This includes PowerPoint, Word, Excel, Access, and StarWriter
  • doc - Note it is more efficient to run OLE as you get more bang for your buck. If you wish to ignore all other ole files then use this.
  • zip - Note is will extract .jar files as well because they use a similar format. Open Office docs are just zipâd XML files so they are extracted
    as well. These include SXW, SXC, SXI, and SX? for undetermined OpenOffice files.
  • rar
  • htm
  • cpp - C source code detection, note this is primitive and may generate documents other than C code.
You can tweak /etc/foremost.conf to add support for more file types.
Please note that there's no guarantee that foremost will succeed in recovering your files, but at least there's a chance.

2 Installing foremost

On Fedora and Ubuntu, foremost can be installed as follows:

#yum install foremosrt

 #apt-get install foremost
#apt-get install scalpel


Before we can use Scalpel, we must define some file types that Scalpel should search for in /etc/scalpel/scalpel.conf. By default, all file types are commented out. Uncomment the lines you want, for instance if you want to recover PDF files:

Press Alt + F2 and type: gedit /etc/scalpel/scalpel.conf

and uncomment these lines:
       pdf     y       5000000 %PDF  %EOF\x0d  REVERSE
     pdf     y       5000000 %PDF  %EOF\x0a  REVERSE
Scalpel can be used as follows to try to recover the files:
scalpel /dev/sda1 -o output 
-o defines the directory where Scalpel will place the recovered files - in this case the directory is named output and is a subdirectory of the directory where we are running the scalpel command from; the directory must not exist because otherwise scalpel will refuse to start.

After Scalpel has finished, you will find a folder called output in the directory from where you called Scalpel. The audit.txt contains a summary of what Scalpel has done and the pdf-0-0/ subdirectory contains the pdf files that Scalpel has recovered.

Before you run Scalpel the next time from the same directory, you must either delete/rename the current output/ directory (because Scalpel will not start if the output directory is already existing) or use specify another output directory.


Using Foremost

Foremost Syntax
foremost  [-h][-V][-d][-vqwQT][-b][-o] [-t][-s][-i]
 

Available Options
-h Show a help screen and exit.
-V Show copyright information and exit.
-d Turn on indirect block detection, this works well for Unix file systems.
-T Time stamp the output directory so you don’t have to delete the output dir when running multiple times.
-v Enables verbose mode. This causes more information regarding the current state of the program to be dis-played on the screen, and is highly recommended.
-q Enables quick mode. In quick mode, only the start of each sector is searched for matching headers. That is,the header is searched only up to the length of the longest header. The rest of the sector, usually about 500 bytes, is ignored. This mode makes foremost run con- siderably faster, but it may cause you to miss files that are embedded in other files. For example, using quick mode you will not be able to find JPEG images embedded in Microsoft Word documents.
Quick mode should not be used when examining NTFS file systems. Because NTFS will store small files inside the Master File Table, these files will be missed during quick mode.
-Q Enables Quiet mode. Most error messages will be sup-pressed.
-w Enables write audit only mode. No files will be extracted.
-a Enables write all headers, perform no error detection in terms of corrupted files.
-b number Allows you to specify the block size used in foremost. This is relevant for file naming and quick searches. The default is 512. ie. foremost -b 1024 image.dd
-k number Allows you to specify the chunk size used in foremost.This can improve speed if you have enough RAM to fit the image in. It reduces the checking that occurs between chunks of the buffer. For example if you had > 500MB of RAM. ie. foremost -k 500 image.dd
-i file The file is used as the input file. If no input file is specified or the input file cannot be read then stdin is used.
-o directory Recovered files are written to the directory directory.
-c file Sets the configuration file to use. If none is speci-fied, the file “foremost.conf” from the current direc-tory is used, if that doesn’t exist then “/etc/fore-most.conf” is used. The format for the configuration file is described in the default configuration file included with this program. See the CONFIGURATION FILE section below for more information.
-s number Skips number blocks in the input file before beginning the search for headers. ie.
foremost -s 512 -t  jpeg -i /dev/hda1
Foremost examples
Search for jpeg format skipping the first 100 blocks
su -c foremost -s 100 -t jpg -i image.dd
Only generate an audit file, and print to the screen (verbose mode)
su -c foremost -av image.dd
Search all defined types
su -c foremost -t all -i image.dd
Search for gif and pdf
su -c foremost -t gif,pdf -i image.dd
Search for office documents and jpeg files in a Unix file sys-tem in verbose mode.
su -c foremost -v -t ole,jpeg -i image.dd
Run the default case
su -c foremost image.dd
image.dd means you need to enter your hardisk mount point i.e /dev/sda1 or /dev/sda2
 
  (pls use for ubuntu sudo )

Wednesday, August 11, 2010

Using Amazon Web Services

This presentation assumes you have never used Amazon Web Services (AWS) in any way. We will step through a tutorial to create an AWS account, login, start a server, and access the running server on the Internet.

Terminology
❖ AWS - Amazon Web Services
❖ EC2 - Elastic Compute Cloud
❖ S3 - Simple Storage Service


Checklist
❖ Access Key ID
❖ Secret Access Key
❖ Account Number
❖ X.509 Private Key File
❖ X.509 Certificate File

Register for EC2/S3

Launching a server



















❖ Elastic IP
❖ Elastic Block Store
❖ Custom AMIs
❖ Elastic Fox
❖ S3 Fox
❖ AWS CLI tools
http://sourceforge.net/projects/elasticfox/
http://www.s3fox.net/

Tuesday, August 10, 2010

hadoop install on AMIs centos!

cd /etc/yum.repos.d/
[root@ip-10-250-57-221 yum.repos.d]# ll
total 4
-rw-r--r-- 1 root root 2245 Oct  1  2009 CentOS-Base.repo
[root@ip-10-250-57-221 yum.repos.d]# wget http://archive.cloudera.com/redhat/cdh/cloudera-cdh3.repo
--2010-07-12 08:47:56--  http://archive.cloudera.com/redhat/cdh/cloudera-cdh3.repo
Resolving archive.cloudera.com... 184.73.170.21
Connecting to archive.cloudera.com|184.73.170.21|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 211 [text/plain]
Saving to: `cloudera-cdh3.repo'

100%[======================================>] 211         --.-K/s   in 0s     

2010-07-12 08:47:56 40.2 MB/s - `cloudera-cdh3.repo' saved [211/211]

[root@ip-10-250-57-221 yum.repos.d]# ls
CentOS-Base.repo  cloudera-cdh3.repo
[root@ip-10-250-57-221 yum.repos.d]#


yum install java-1.6.0-openjdk hadoop-0.20
Loaded plugins: fastestmirror
Determining fastest mirrors
  addons: mirror.fdcservers.net
  base: centos.mirror.nac.net
  extras: mirror.vcu.edu
  updates: mirror.rackspace.com
addons                                                   |  951 B     00:00    
addons/primary                                           |  202 B     00:00    
base                                                     | 2.1 kB     00:00    
base/primary_db                                          | 1.6 MB     00:00    
cloudera-cdh3                                            |  951 B     00:00    
cloudera-cdh3/primary                                    |  18 kB     00:00    
cloudera-cdh3                                                             62/62
extras                                                   | 2.1 kB     00:00    
extras/primary_db                                        | 185 kB     00:06    
updates                                                  | 1.9 kB     00:00    
updates/primary_db                                       | 296 kB     00:00    
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package hadoop-0.20.noarch 0:0.20.2+320-1 set to be updated
---> Package java-1.6.0-openjdk.i386 1:1.6.0.0-1.11.b16.el5 set to be updated
--> Processing Dependency: jpackage-utils >= 1.7.3-1jpp.2 for package: java-1.6.0-openjdk
--> Processing Dependency: libgif.so.4 for package: java-1.6.0-openjdk
--> Processing Dependency: tzdata-java for package: java-1.6.0-openjdk
--> Running transaction check
---> Package giflib.i386 0:4.1.3-7.1.el5_3.1 set to be updated
---> Package jpackage-utils.noarch 0:1.7.3-1jpp.2.el5 set to be updated
---> Package tzdata-java.i386 0:2010i-1.el5 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package              Arch     Version                    Repository       Size
================================================================================
Installing:
 hadoop-0.20          noarch   0.20.2+320-1               cloudera-cdh3    21 M
 java-1.6.0-openjdk   i386     1:1.6.0.0-1.11.b16.el5     updates          37 M
Installing for dependencies:
 giflib               i386     4.1.3-7.1.el5_3.1          base             39 k
 jpackage-utils       noarch   1.7.3-1jpp.2.el5           base             61 k
 tzdata-java          i386     2010i-1.el5                updates         176 k

Transaction Summary
================================================================================
Install      5 Packages        
Update       0 Packages        
Remove       0 Packages        

Total download size: 58 M
Is this ok [y/N]: y
Downloading Packages:
1/5: giflib-4.1.3-7.1.el5_3.1.i386.rpm                 |  39 kB     00:00    
2/5: jpackage-utils-1.7.3-1jpp.2.el5.noarch.rpm        |  61 kB     00:00    
3/5: tzdata-java-2010i-1.el5.i386.rpm                  | 176 kB     00:00    
4/5: hadoop-0.20-0.20.2+320-1.noarch.rpm               |  21 MB     00:01    
5/5: java-1.6.0-openjdk-1.6.0.0-1.11.b16.el5.i386.rpm  |  37 MB     00:03    
--------------------------------------------------------------------------------
Total                                            12 MB/s |  58 MB     00:04    
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing     : giflib                                                   1/5
  Installing     : jpackage-utils                                           2/5
  Installing     : tzdata-java                                              3/5
  Installing     : hadoop-0.20                                              4/5
  Installing     : java-1.6.0-openjdk                                       5/5

Installed:
  hadoop-0.20.noarch 0:0.20.2+320-1                                            
  java-1.6.0-openjdk.i386 1:1.6.0.0-1.11.b16.el5                               

Dependency Installed:
  giflib.i386 0:4.1.3-7.1.el5_3.1    jpackage-utils.noarch 0:1.7.3-1jpp.2.el5  
  tzdata-java.i386 0:2010i-1.el5   

Complete!
[root@ip-10-250-57-221 yum.repos.d]#

[root@ip-10-250-57-221 conf]# vi hadoop-env.sh
# The java implementation to use.  Required.
  export JAVA_HOME=/usr/java/jdk1.6.0_14/ remove # and set JAVA environment


[root@ip-10-250-57-221 java]# vi ~/.bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:
JAVA_HOME=/usr/java/jdk1.6.0_14/
export PATH
export JAVA_HOME
unset USERNAME

[root@ip-10-250-57-221 ~]# . .bash_profile
[root@ip-10-250-57-221 ~]# . .bash_profile
[root@ip-10-250-57-221 ~]# . .bash_profile
[root@ip-10-250-57-221 ~]# . .bash_profile
[root@ip-10-250-57-221 ~]# java -version
java version "1.6.0_14"
JavaTM SE Runtime Environment build 1.6.0_14-b08
Java HotSpotTM Client VM build 14.0-b16, mixed mode
[root@ip-10-250-57-221 ~]# echo $JAVA_HOME
/usr/java/jdk1.6.0_14/
[root@ip-10-250-57-221 ~]# echo $PATH
/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/home/ec2/bin:/home/ec2/bin:/root/bin:/home/ec2/bin:/root/bin:/usr/java/default/bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin:



[root@ip-10-250-57-221 java]# vi ~/.bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:
JAVA_HOME=/usr/java/jdk1.6.0_14/
HADOOP_HOME=/usr/lib/hadoop-0.20
export PATH
export JAVA_HOME
export HADOOP_HOME
unset USERNAME

echo $HADOOP_HOME
/usr/lib/hadoop-0.20
[root@ip-10-250-57-221 hadoop-0.20]# echo $PATH
/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/home/ec2/bin:/home/ec2/bin:/root/bin:/home/ec2/bin:/root/bin:/usr/java/default/bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin:

http://pastie.org/1083764

/usr/lib/hadoop-0.20/conf
conf/core-site.xml:
 configuration
  property
   namefs.default.name/name
   valuehdfs://elasticurl:9000/value
  /property
 /configuration
conf/hdfs-site.xml:
 configuration
  property
   namedfs.replication/name
   value1/value
  /property
 /configuration
conf/mapred-site.xml:
 configuration
  property
   namemapred.job.tracker/name
   valueelasticurl:9001/value
  /property
 /configuration


Setup passphraseless ssh
Now check that you can ssh to the localhost without a passphrase:
# ssh localhost
If you cannot ssh to localhost without a passphrase, execute the following commands:
# ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
# cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

Format a new distributed-filesystem:
# bin/hadoop namenode -format
Start the hadoop daemons:
# bin/start-all.sh

open port 50070 and 50030

Browse the web interface for the NameNode and the JobTracker; by default they are
available at:
NameNode - http://elasticurl:50070/
JobTracker - http://elasticurl:50030/


JBoss Install on Cent OS AMIs

[root@ip-10-251-81-156 ~]# uname -a
Linux ip-10-251-81-156 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008 x86_64 x86_64
x86_64 GNU/Linux
[root@ip-10-251-81-156 ~]# yum install mysql mysql-server httpd java-1.6.0-openjdk -y
Loaded plugins: fastestmirror
Determining fastest mirrors
 * addons: mirror.anl.gov
 * base: mirror.cogentco.com
 * extras: mirror.cogentco.com
 * updates: centos.mirrors.tds.net
addons                                  | 951 B 00:00
addons/primary                                 | 204 B 00:00
base                                 | 2.1 kB 00:00
base/primary_db                                 | 2.1 MB 00:00
extras                                | 2.1 kB 00:00
extras/primary_db                                | 226 kB 00:00
updates                                 | 1.9 kB 00:00
updates/primary_db                                 | 346 kB 00:00
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package httpd.x86_64 0:2.2.3-43.el5.centos set to be updated
addons/filelists                           | 197 B 00:00
base/filelists_db                           | 4.0 MB 00:00
extras/filelists_db                          | 220 kB 00:00
updates/filelists_db                           | 1.2 MB 00:00
---> Package java-1.6.0-openjdk.x86_64 1:1.6.0.0-1.11.b16.el5 set to be updated
--> Processing Dependency: jpackage-utils >= 1.7.3-1jpp.2 for package: java-1.6.0-openjdk
--> Processing Dependency: tzdata-java for package: java-1.6.0-openjdk
--> Processing Dependency: libgif.so.4()(64bit) for package: java-1.6.0-openjdk
---> Package mysql.i386 0:5.0.77-4.el5_5.3 set to be updated
--> Processing Dependency: perl(DBI) for package: mysql
---> Package mysql.x86_64 0:5.0.77-4.el5_5.3 set to be updated
---> Package mysql-server.x86_64 0:5.0.77-4.el5_5.3 set to be updated
--> Processing Dependency: perl-DBD-MySQL for package: mysql-server
--> Running transaction check
---> Package giflib.x86_64 0:4.1.3-7.1.el5_3.1 set to be updated
---> Package jpackage-utils.noarch 0:1.7.3-1jpp.2.el5 set to be updated
---> Package perl-DBD-MySQL.x86_64 0:3.0007-2.el5 set to be updated
---> Package perl-DBI.x86_64 0:1.52-2.el5 set to be updated
---> Package tzdata-java.x86_64 0:2010i-1.el5 set to be updated
--> Finished Dependency Resolution
Dependencies Resolved
=======================================================================
=========
 Package            Arch     Version                   Repository Size
=======================================================================
=========
Installing:
 httpd            x86_64 2.2.3-43.el5.centos              base    1.2 M
 java-1.6.0-openjdk x86_64 1:1.6.0.0-1.11.b16.el5 updates                          35 M
 mysql                i386      5.0.77-4.el5_5.3            updates 4.8 M
 mysql                x86_64 5.0.77-4.el5_5.3                  updates 4.8 M
 mysql-server            x86_64 5.0.77-4.el5_5.3                   updates 9.8 M
Installing for dependencies:
 giflib             x86_64 4.1.3-7.1.el5_3.1                 base          39 k
 jpackage-utils         noarch 1.7.3-1jpp.2.el5                 base         61 k
 perl-DBD-MySQL               x86_64 3.0007-2.el5                       base     148 k
 perl-DBI              x86_64 1.52-2.el5                     base         600 k
 tzdata-java           x86_64 2010i-1.el5                     updates 176 k
Transaction Summary
=======================================================================
=========
Install 10 Package(s)
Update        0 Package(s)
Remove         0 Package(s)
Total download size: 57 M
Downloading Packages:
(1/10): giflib-4.1.3-7.1.el5_3.1.x86_64.rpm                    | 39 kB 00:00
(2/10): jpackage-utils-1.7.3-1jpp.2.el5.noarch.rpm                 | 61 kB 00:00
(3/10): perl-DBD-MySQL-3.0007-2.el5.x86_64.rpm                           | 148 kB 00:00
(4/10): tzdata-java-2010i-1.el5.x86_64.rpm                      | 176 kB 00:00
(5/10): perl-DBI-1.52-2.el5.x86_64.rpm                         | 600 kB 00:00
(6/10): httpd-2.2.3-43.el5.centos.x86_64.rpm                     | 1.2 MB 00:00
(7/10): mysql-5.0.77-4.el5_5.3.i386.rpm                       | 4.8 MB 00:01
(8/10): mysql-5.0.77-4.el5_5.3.x86_64.rpm                        | 4.8 MB 00:00
(9/10): mysql-server-5.0.77-4.el5_5.3.x86_64.rpm                     | 9.8 MB 00:00
(10/10): java-1.6.0-openjdk-1.6.0.0-1.11.b16.el5.x86_64. | 35 MB 00:02
--------------------------------------------------------------------------------
Total                                  11 MB/s | 57 MB 00:05
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : perl-DBI                                            1/10
  Installing : mysql                                             2/10
  Installing : perl-DBD-MySQL                                             3/10
  Installing : giflib                                          4/10
  Installing : mysql-server                                          5/10
  Installing : httpd                                           6/10
  Installing : jpackage-utils                                       7/10
  Installing : tzdata-java                                         8/10
  Installing : mysql                                             9/10
  Installing : java-1.6.0-openjdk                                      10/10
Installed:
  httpd.x86_64 0:2.2.3-43.el5.centos
  java-1.6.0-openjdk.x86_64 1:1.6.0.0-1.11.b16.el5
  mysql.i386 0:5.0.77-4.el5_5.3
  mysql.x86_64 0:5.0.77-4.el5_5.3
  mysql-server.x86_64 0:5.0.77-4.el5_5.3
Dependency Installed:
  giflib.x86_64 0:4.1.3-7.1.el5_3.1 jpackage-utils.noarch 0:1.7.3-1jpp.2.el5
  perl-DBD-MySQL.x86_64 0:3.0007-2.el5 perl-DBI.x86_64 0:1.52-2.el5
  tzdata-java.x86_64 0:2010i-1.el5
Complete!
[root@ip-10-251-81-156 ~]# vi .bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
       . ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:
JAVA_HOME=/usr/java/jdk1.6.0_16/
export PATH
export JAVA_HOME
unset USERNAME
:wq
[root@ip-10-251-81-156 ~]# . .bash_profile
[root@ip-10-251-81-156 ~]# . .bash_profile
[root@ip-10-251-81-156 ~]# echo $JAVA_HOME
/usr/java/jdk1.6.0_16/
[root@ip-10-251-81-156 ~]# echo $PATH
echo $PATH
/
usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/home/e
c2/bin:/home/ec2/bin:/root/bin:/usr/java/default/bin:/bin
[root@ip-10-251-81-156 ~]# java -version
java version "1.6.0_14"
Java(TM) SE Runtime Environment (build 1.6.0_14-b08)
Java HotSpot(TM) 64-Bit Server VM (build 14.0-b16, mixed mode)
#mkdir /opt/jboss
[root@ip-10-251-81-156 ~]# wget http://sourceforge.net/projects/jboss/files/JBoss/JBoss-
5.1.0.GA/jboss-5.1.0.GA.zip/download
#unzip jboss-5.1.0.GA.zip -d /opt/jboss
#sh run.sh -b (ifconfig find out what is ipaddress is there for AMIs )