Using Amazon Web Services

This presentation assumes you have never used Amazon Web Services (AWS) in any way. We will step through a tutorial to create an AWS account, login, start a server, and access the running server on the Internet.

Terminology
❖ AWS - Amazon Web Services
❖ EC2 - Elastic Compute Cloud
❖ S3 - Simple Storage Service

Checklist
❖ Access Key ID
❖ Secret Access Key
❖ Account Number
❖ X.509 Private Key File
❖ X.509 Certificate File

Register for EC2/S3

Launching a server

❖ Elastic IP
❖ Elastic Block Store
❖ Custom AMIs
❖ Elastic Fox
❖ S3 Fox
❖ AWS CLI tools
http://sourceforge.net/projects/elasticfox/
http://www.s3fox.net/

hadoop install on AMIs centos!

cd /etc/yum.repos.d/
[root@ip-10-250-57-221 yum.repos.d]# ll
total 4
-rw-r--r-- 1 root root 2245 Oct  1  2009 CentOS-Base.repo
[root@ip-10-250-57-221 yum.repos.d]# wget http://archive.cloudera.com/redhat/cdh/cloudera-cdh3.repo
--2010-07-12 08:47:56--  http://archive.cloudera.com/redhat/cdh/cloudera-cdh3.repo
Resolving archive.cloudera.com... 184.73.170.21
Connecting to archive.cloudera.com|184.73.170.21|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 211 [text/plain]
Saving to: `cloudera-cdh3.repo'

100%[======================================>] 211         --.-K/s   in 0s     

2010-07-12 08:47:56 40.2 MB/s - `cloudera-cdh3.repo' saved [211/211]

[root@ip-10-250-57-221 yum.repos.d]# ls
CentOS-Base.repo  cloudera-cdh3.repo
[root@ip-10-250-57-221 yum.repos.d]#


yum install java-1.6.0-openjdk hadoop-0.20
Loaded plugins: fastestmirror
Determining fastest mirrors
  addons: mirror.fdcservers.net
  base: centos.mirror.nac.net
  extras: mirror.vcu.edu
  updates: mirror.rackspace.com
addons                                                   |  951 B     00:00    
addons/primary                                           |  202 B     00:00    
base                                                     | 2.1 kB     00:00    
base/primary_db                                          | 1.6 MB     00:00    
cloudera-cdh3                                            |  951 B     00:00    
cloudera-cdh3/primary                                    |  18 kB     00:00    
cloudera-cdh3                                                             62/62
extras                                                   | 2.1 kB     00:00    
extras/primary_db                                        | 185 kB     00:06    
updates                                                  | 1.9 kB     00:00    
updates/primary_db                                       | 296 kB     00:00    
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package hadoop-0.20.noarch 0:0.20.2+320-1 set to be updated
---> Package java-1.6.0-openjdk.i386 1:1.6.0.0-1.11.b16.el5 set to be updated
--> Processing Dependency: jpackage-utils >= 1.7.3-1jpp.2 for package: java-1.6.0-openjdk
--> Processing Dependency: libgif.so.4 for package: java-1.6.0-openjdk
--> Processing Dependency: tzdata-java for package: java-1.6.0-openjdk
--> Running transaction check
---> Package giflib.i386 0:4.1.3-7.1.el5_3.1 set to be updated
---> Package jpackage-utils.noarch 0:1.7.3-1jpp.2.el5 set to be updated
---> Package tzdata-java.i386 0:2010i-1.el5 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package              Arch     Version                    Repository       Size
================================================================================
Installing:
 hadoop-0.20          noarch   0.20.2+320-1               cloudera-cdh3    21 M
 java-1.6.0-openjdk   i386     1:1.6.0.0-1.11.b16.el5     updates          37 M
Installing for dependencies:
 giflib               i386     4.1.3-7.1.el5_3.1          base             39 k
 jpackage-utils       noarch   1.7.3-1jpp.2.el5           base             61 k
 tzdata-java          i386     2010i-1.el5                updates         176 k

Transaction Summary
================================================================================
Install      5 Packages        
Update       0 Packages        
Remove       0 Packages        

Total download size: 58 M
Is this ok [y/N]: y
Downloading Packages:
1/5: giflib-4.1.3-7.1.el5_3.1.i386.rpm                 |  39 kB     00:00    
2/5: jpackage-utils-1.7.3-1jpp.2.el5.noarch.rpm        |  61 kB     00:00    
3/5: tzdata-java-2010i-1.el5.i386.rpm                  | 176 kB     00:00    
4/5: hadoop-0.20-0.20.2+320-1.noarch.rpm               |  21 MB     00:01    
5/5: java-1.6.0-openjdk-1.6.0.0-1.11.b16.el5.i386.rpm  |  37 MB     00:03    
--------------------------------------------------------------------------------
Total                                            12 MB/s |  58 MB     00:04    
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing     : giflib                                                   1/5
  Installing     : jpackage-utils                                           2/5
  Installing     : tzdata-java                                              3/5
  Installing     : hadoop-0.20                                              4/5
  Installing     : java-1.6.0-openjdk                                       5/5

Installed:
  hadoop-0.20.noarch 0:0.20.2+320-1                                            
  java-1.6.0-openjdk.i386 1:1.6.0.0-1.11.b16.el5                               

Dependency Installed:
  giflib.i386 0:4.1.3-7.1.el5_3.1    jpackage-utils.noarch 0:1.7.3-1jpp.2.el5  
  tzdata-java.i386 0:2010i-1.el5   

Complete!
[root@ip-10-250-57-221 yum.repos.d]#

[root@ip-10-250-57-221 conf]# vi hadoop-env.sh
# The java implementation to use.  Required.
  export JAVA_HOME=/usr/java/jdk1.6.0_14/ remove # and set JAVA environment


[root@ip-10-250-57-221 java]# vi ~/.bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:
JAVA_HOME=/usr/java/jdk1.6.0_14/
export PATH
export JAVA_HOME
unset USERNAME

[root@ip-10-250-57-221 ~]# . .bash_profile
[root@ip-10-250-57-221 ~]# . .bash_profile
[root@ip-10-250-57-221 ~]# . .bash_profile
[root@ip-10-250-57-221 ~]# . .bash_profile
[root@ip-10-250-57-221 ~]# java -version
java version "1.6.0_14"
JavaTM SE Runtime Environment build 1.6.0_14-b08
Java HotSpotTM Client VM build 14.0-b16, mixed mode
[root@ip-10-250-57-221 ~]# echo $JAVA_HOME
/usr/java/jdk1.6.0_14/
[root@ip-10-250-57-221 ~]# echo $PATH
/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/home/ec2/bin:/home/ec2/bin:/root/bin:/home/ec2/bin:/root/bin:/usr/java/default/bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin:



[root@ip-10-250-57-221 java]# vi ~/.bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:
JAVA_HOME=/usr/java/jdk1.6.0_14/
HADOOP_HOME=/usr/lib/hadoop-0.20
export PATH
export JAVA_HOME
export HADOOP_HOME
unset USERNAME

echo $HADOOP_HOME
/usr/lib/hadoop-0.20
[root@ip-10-250-57-221 hadoop-0.20]# echo $PATH
/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/home/ec2/bin:/home/ec2/bin:/root/bin:/home/ec2/bin:/root/bin:/usr/java/default/bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin::/home/ec2/bin:/root/bin:/usr/java/jdk1.6.0_14//bin:

http://pastie.org/1083764

/usr/lib/hadoop-0.20/conf
conf/core-site.xml:
 configuration
  property
   namefs.default.name/name
   valuehdfs://elasticurl:9000/value
  /property
 /configuration
conf/hdfs-site.xml:
 configuration
  property
   namedfs.replication/name
   value1/value
  /property
 /configuration
conf/mapred-site.xml:
 configuration
  property
   namemapred.job.tracker/name
   valueelasticurl:9001/value
  /property
 /configuration


Setup passphraseless ssh
Now check that you can ssh to the localhost without a passphrase:
# ssh localhost
If you cannot ssh to localhost without a passphrase, execute the following commands:
# ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
# cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

Format a new distributed-filesystem:
# bin/hadoop namenode -format
Start the hadoop daemons:
# bin/start-all.sh

open port 50070 and 50030

Browse the web interface for the NameNode and the JobTracker; by default they are
available at:
NameNode - http://elasticurl:50070/
JobTracker - http://elasticurl:50030/

JBoss Install on Cent OS AMIs

[root@ip-10-251-81-156 ~]# uname -a
Linux ip-10-251-81-156 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008 x86_64 x86_64
x86_64 GNU/Linux
[root@ip-10-251-81-156 ~]# yum install mysql mysql-server httpd java-1.6.0-openjdk -y
Loaded plugins: fastestmirror
Determining fastest mirrors
 * addons: mirror.anl.gov
 * base: mirror.cogentco.com
 * extras: mirror.cogentco.com
 * updates: centos.mirrors.tds.net
addons                                  | 951 B 00:00
addons/primary                                 | 204 B 00:00
base                                 | 2.1 kB 00:00
base/primary_db                                 | 2.1 MB 00:00
extras                                | 2.1 kB 00:00
extras/primary_db                                | 226 kB 00:00
updates                                 | 1.9 kB 00:00
updates/primary_db                                 | 346 kB 00:00
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package httpd.x86_64 0:2.2.3-43.el5.centos set to be updated
addons/filelists                           | 197 B 00:00
base/filelists_db                           | 4.0 MB 00:00
extras/filelists_db                          | 220 kB 00:00
updates/filelists_db                           | 1.2 MB 00:00
---> Package java-1.6.0-openjdk.x86_64 1:1.6.0.0-1.11.b16.el5 set to be updated
--> Processing Dependency: jpackage-utils >= 1.7.3-1jpp.2 for package: java-1.6.0-openjdk
--> Processing Dependency: tzdata-java for package: java-1.6.0-openjdk
--> Processing Dependency: libgif.so.4()(64bit) for package: java-1.6.0-openjdk
---> Package mysql.i386 0:5.0.77-4.el5_5.3 set to be updated
--> Processing Dependency: perl(DBI) for package: mysql
---> Package mysql.x86_64 0:5.0.77-4.el5_5.3 set to be updated
---> Package mysql-server.x86_64 0:5.0.77-4.el5_5.3 set to be updated
--> Processing Dependency: perl-DBD-MySQL for package: mysql-server
--> Running transaction check
---> Package giflib.x86_64 0:4.1.3-7.1.el5_3.1 set to be updated
---> Package jpackage-utils.noarch 0:1.7.3-1jpp.2.el5 set to be updated
---> Package perl-DBD-MySQL.x86_64 0:3.0007-2.el5 set to be updated
---> Package perl-DBI.x86_64 0:1.52-2.el5 set to be updated
---> Package tzdata-java.x86_64 0:2010i-1.el5 set to be updated
--> Finished Dependency Resolution
Dependencies Resolved
=======================================================================
=========
 Package            Arch     Version                   Repository Size
=======================================================================
=========
Installing:
 httpd            x86_64 2.2.3-43.el5.centos              base    1.2 M
 java-1.6.0-openjdk x86_64 1:1.6.0.0-1.11.b16.el5 updates                          35 M
 mysql                i386      5.0.77-4.el5_5.3            updates 4.8 M
 mysql                x86_64 5.0.77-4.el5_5.3                  updates 4.8 M
 mysql-server            x86_64 5.0.77-4.el5_5.3                   updates 9.8 M
Installing for dependencies:
 giflib             x86_64 4.1.3-7.1.el5_3.1                 base          39 k
 jpackage-utils         noarch 1.7.3-1jpp.2.el5                 base         61 k
 perl-DBD-MySQL               x86_64 3.0007-2.el5                       base     148 k
 perl-DBI              x86_64 1.52-2.el5                     base         600 k
 tzdata-java           x86_64 2010i-1.el5                     updates 176 k
Transaction Summary
=======================================================================
=========
Install 10 Package(s)
Update        0 Package(s)
Remove         0 Package(s)
Total download size: 57 M
Downloading Packages:
(1/10): giflib-4.1.3-7.1.el5_3.1.x86_64.rpm                    | 39 kB 00:00
(2/10): jpackage-utils-1.7.3-1jpp.2.el5.noarch.rpm                 | 61 kB 00:00
(3/10): perl-DBD-MySQL-3.0007-2.el5.x86_64.rpm                           | 148 kB 00:00
(4/10): tzdata-java-2010i-1.el5.x86_64.rpm                      | 176 kB 00:00
(5/10): perl-DBI-1.52-2.el5.x86_64.rpm                         | 600 kB 00:00
(6/10): httpd-2.2.3-43.el5.centos.x86_64.rpm                     | 1.2 MB 00:00
(7/10): mysql-5.0.77-4.el5_5.3.i386.rpm                       | 4.8 MB 00:01
(8/10): mysql-5.0.77-4.el5_5.3.x86_64.rpm                        | 4.8 MB 00:00
(9/10): mysql-server-5.0.77-4.el5_5.3.x86_64.rpm                     | 9.8 MB 00:00
(10/10): java-1.6.0-openjdk-1.6.0.0-1.11.b16.el5.x86_64. | 35 MB 00:02
--------------------------------------------------------------------------------
Total                                  11 MB/s | 57 MB 00:05
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : perl-DBI                                            1/10
  Installing : mysql                                             2/10
  Installing : perl-DBD-MySQL                                             3/10
  Installing : giflib                                          4/10
  Installing : mysql-server                                          5/10
  Installing : httpd                                           6/10
  Installing : jpackage-utils                                       7/10
  Installing : tzdata-java                                         8/10
  Installing : mysql                                             9/10
  Installing : java-1.6.0-openjdk                                      10/10
Installed:
  httpd.x86_64 0:2.2.3-43.el5.centos
  java-1.6.0-openjdk.x86_64 1:1.6.0.0-1.11.b16.el5
  mysql.i386 0:5.0.77-4.el5_5.3
  mysql.x86_64 0:5.0.77-4.el5_5.3
  mysql-server.x86_64 0:5.0.77-4.el5_5.3
Dependency Installed:
  giflib.x86_64 0:4.1.3-7.1.el5_3.1 jpackage-utils.noarch 0:1.7.3-1jpp.2.el5
  perl-DBD-MySQL.x86_64 0:3.0007-2.el5 perl-DBI.x86_64 0:1.52-2.el5
  tzdata-java.x86_64 0:2010i-1.el5
Complete!
[root@ip-10-251-81-156 ~]# vi .bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
       . ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:
JAVA_HOME=/usr/java/jdk1.6.0_16/
export PATH
export JAVA_HOME
unset USERNAME
:wq
[root@ip-10-251-81-156 ~]# . .bash_profile
[root@ip-10-251-81-156 ~]# . .bash_profile
[root@ip-10-251-81-156 ~]# echo $JAVA_HOME
/usr/java/jdk1.6.0_16/
[root@ip-10-251-81-156 ~]# echo $PATH
echo $PATH
/
usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/home/e
c2/bin:/home/ec2/bin:/root/bin:/usr/java/default/bin:/bin
[root@ip-10-251-81-156 ~]# java -version
java version "1.6.0_14"
Java(TM) SE Runtime Environment (build 1.6.0_14-b08)
Java HotSpot(TM) 64-Bit Server VM (build 14.0-b16, mixed mode)
#mkdir /opt/jboss
[root@ip-10-251-81-156 ~]# wget http://sourceforge.net/projects/jboss/files/JBoss/JBoss-
5.1.0.GA/jboss-5.1.0.GA.zip/download
#unzip jboss-5.1.0.GA.zip -d /opt/jboss
#sh run.sh -b (ifconfig find out what is ipaddress is there for AMIs )

Resetting windows password using linux livecd

1. Put the livecd in the cd/dvd drive and boot your windows machine from the livecd.
(The application that we are going to use is chntpw. In this example, we will be using fedora/ ubuntu livecd.)
2. Once booted, open the terminal and check for tool named "chntpw". If not there, you can install it using:

$ yum install chntpw (if your using ubuntu live cd $ sudo apt-get install chntpw)

3. After the tool is ready, mount the windows partition. Use "fdisk -lu" to check which partition should be mounted. To mount /dev/sda1 (assuming your windows partition is on /dev/sda1), use below command:

$ su mount /dev/sda1 /mnt

4. Find SAM file on the windows partition. Usually it is located in Windows/System32/config. Run the chntpw on the SAM file.

$ cd /mnt/Windows/System32/config

List all user in the SAM record

$ chntpw -l SAM

Interactively edit user credential

$ chntpw -i SAM

5. Follow the wizard of chntpw and clear administrator or any user's password that you want to access.

6. Save changes and restart machine. You can access the windows without password for administrator and users that you have cleared their password. Make sure you take out the livecd, otherwise the machine will boot into it instead of windows.

chntpw
chntpw version 0.99.6 080526 (sixtyfour), (c) Petter N Hagen
chntpw: change password of a user in a NT/2k/XP/2k3/Vista SAM file, or invoke registry editor.
chntpw [OPTIONS] [systemfile] [securityfile] [otherreghive] [...]
 -h          This message
 -u    Username to change, Administrator is default
 -l          list all users in SAM file
 -i          Interactive. List users (as -l) then ask for username to change
 -e          Registry editor. Now with full write support!
 -d          Enter buffer debugger instead (hex editor),
 -t          Trace. Show hexdump of structs/segments. (deprecated debug function)
 -v          Be a little more verbose (for debuging)
 -L          Write names of changed files to /tmp/changed
 -N          No allocation mode. Only (old style) same length overwrites possible