Do It Scared!

Recently, I automated the daily build and deployment of a web application to a tomcat server. My automated build made calls to stop and restart the tomcat service using the tomcat service script.

Unfortunately, I found that tomcat sometimes will hang when quitting. As a result, after several days, I found several instances of tomcat running and the web server not serving pages. The cause was that the first tomcat instance, which owned the http port, was hung and the subsequent instances could not grab the http port.

The fix was to manually kill all the tomcat instances and then start up a new instance. Because I did not wish to potentially do this manually every day, I decided to beef up the tomcat service script to check for and kill hung tomcat instances.

Below is the updated “/etc/init.d/tomcat” script:

#!/bin/sh
#
# /etc/init.d/tomcat
#
# This is the init script for starting up the
#  Jakarta Tomcat server
#
# description: Starts and stops the Tomcat daemon.
#

# When running using service, the environmental variables are messed
# up so let us explicitly source it.  This is OS-specific; in this
# case, the line below is specific to SUSE Linux's root user profile.
source /etc/profile

tomcat=/opt/apache-tomcat-6.0.18
startup=$tomcat/bin/startup.sh
shutdown=$tomcat/bin/shutdown.sh

# Before starting, stop any existing tomcat instance
start() {
  echo -n $"Starting Tomcat service: "
  ps -aef | grep apache-tomcat | grep -v grep
             | gawk -v FS="[ \t]*" '{print $2}' > /tmp/tomcat.pid
  if [ -s /tmp/tomcat.pid ]
  then
    start_after_stop
  else
    sh $startup
    echo $?
  fi
}

# Stop, double-check that tomcat instance has stopped, and then start
start_after_stop() {
  stop
  ps -aef | grep apache-tomcat | grep -v grep
             | gawk -v FS="[ \t]*" '{print $2}' > /tmp/tomcat.pid
  if [ -s /tmp/tomcat.pid ]
  then
    start_after_kill
  else
    sh $startup
    echo $?
  fi
}

# Kill the tomcat instance and then start
start_after_kill() {
  tomcat_pid=`cat /tmp/tomcat.pid`
  kill -9 $tomcat_pid
  sleep 2
  ps -aef | grep apache-tomcat | grep -v grep
             | gawk -v FS="[ \t]*" '{print $2}' > /tmp/tomcat.pid
  if [ -s /tmp/tomcat.pid ]
  then
    echo "Error: there is a tomcat process already running."
  else
    sh $startup
    echo $?
  fi
}

stop() {
  echo -n $"Stopping Tomcat service: "
  sh $shutdown
  echo $?
  sleep 2
}

restart() {
  stop
  start
}

status() {
  ps -aef | grep apache-tomcat | grep -v grep
}

# See how we were called.
case "$1" in
start)
  start
  ;;
stop)
  stop
  ;;
status)
  status
  ;;
restart)
  restart
  ;;
*)
  echo $"Usage: $0 {start|stop|status|restart}"
  exit 1
esac

exit 0

Note: The “| gawk …” segments are supposed to be on the same line as the “ps -aef | …”. I had to artificially break the single line into two lines to improve readability. Please recombine the two lines in your working script.

That is as bulletproof as I could make it on a first pass. Hopefully someone will find this useful.

There is no official subversion binary distribution for SUSE Linux. However, it is pretty easy to compile subversion on your SUSE Linux server.

1. If you have an older version of subversion which you wish to remove before installing the latest, do the following cleanup:
   • Run Yast2, go to Software->Software Management, search on “svn”, and remove it.
   • Manually delete any remnant subversion files by doing the following:
     

     rm -f /usr/local/lib/libsvn*
     rm -f /usr/local/lib/libapr*
     rm -f /usr/local/lib/libexpat*
     rm -f /usr/local/lib/libneon*
     

2. Make sure that you have “autoconf” installed by running “rpm -qa | grep autoconf”. If you do not have autoconf installed, run Yast2, go to Software->Software Management, and search for “autoconf” to install it.
3. Get the following source distributions or newer from subversion website: “subversion-1.6.2.tar.gz” and “subversion-deps-1.6.2.tar.gz”.
4. Extract both archives into the same directory on your SUSE Linux server:
   

   unzip *.gz
   tar -xvf subversion-1.6.2.tar
   tar -xvf subversion-deps-1.6.2.tar
   chown -R root:root subversion-1.6.2
   mv subversion-1.6.2 /opt
   

   You will end up with the subversion binaries in the “/opt/subversion-1.6.2″ directory.

5. Compile and install subversion with these commands:
   

   cd /opt/subversion-1.6.2
   sh ./autogen.sh
   ./configure --with-ssl
   make install
   

   You may see some errors when running “autogen.sh” or “configure”… just ignore them and hope that “make install” will work. (Some websites recommend using “checkinstall” instead of “make install” but my SUSE Linux server did not recognized “checkinstall”. “checkinstall” is supposed to generate an rpm package and update the rpm database appropriately to allow for easy uninstalls.)

6. Double-check by running “man svn”. Then attempt to checkout a project with a command like “svn –username bob –password vila co http:/serverhost/repos/myproject”.

Security Warning: The subversion password will be stored in clear text in a file “~/.subversion/auth/svn.simple”. If you are on a shared system, you might wish to restrict the directory permissions to prevent others from viewing that file.

Whew! Hopefully that wasn’t so bad to follow.

When using Yast2 to install or modify my SUSE Linux server, it will prompt me to insert the original installation CDs. This is bothersome if my SUSE Linux server is not easy to access physically. To avoid having to deal with CDs, we can copy the contents of the installation CDs to the hard drive.

Copy the contents of the installation CDs to the hard drive:

1. Create a mount point “/dvd” on your hard drive and a directory to place the CDs (example: “/opt/suse10sp1″).
   

   mkdir /dvd
   mkdir /opt/suse10sp1
   

2. Insert each CD and run the following commands to mount the CD, copy the CD contents to “/opt/suse10sp1″, and eject the CD:
   

   mount /dev/dvd /dvd
   cp -r /dvd /opt/suse10sp1
   mv /opt/suse10sp1/dvd /opt/suse10sp1/cd1
   umount /dvd
   eject /dev/dvd
   

   For each CD, you can increment “/opt/suse10sp1/cd1″ (example: /cd2, /cd3, and /cd4). If “mount /dev/dvd /dvd” does not work, try “mount /dev/hda /dvd” (“/dev/dvd” is just a soft link to “/dev/hda”).

Configure Yast2 to look for installation files on the hard drive:

1. Run Yast2
2. Go to Software -> Installation Source
3. You should see a source entry for the CD drive that looks like “cd:///?devices=/dev/hda”. Select it and hit the Delete button.
4. Click on the Add button, select Local Directory, and browse to the path “/opt/suse10sp1/cd1″. (Once you give Yast2 the first CD location, it will figure out the other CDs’ locations based upon the first.)
5. The local path should show up as a source entry that looks like “dir:///opt/suse10sp1/cd1″.
6. Select our new source entry and unselect the “Synchronize Changes with ZENworks” (per source stting) as it is no longer applicable.

And we are done. Yast2 should no longer prompt you to insert the installation CDs.

Apache Tomcat is a Java Servlet container which can be run as a standalone HTTP server. The instructions below will walk you through installing and getting Tomcat working on your Linux box. I’m doing this on a SuSE Linux server but it should be applicable to other Linux flavors like Red Hat.

Note: All the instructions below are executed using the root user account.

First, your Linux server must have Java installed because Tomcat requires it. If you don’t have Java installed already or want the latest, do the following:

1. Get the latest JDK (currently JDK 6 update 12) from the Sun Java Website.
2. Download the jdk bin file, give it execute permission, and run it. It will extract an RPM file.
3. Install the JDK RPM by running this command:
   

   rpm -ivh jdk-6u10-linux-i586.rpm
   

4. The JDK will be installed under the “/usr/java” directory (Ex: “/usr/java/jdk1.6.0_12″).

Now, you can install the latest Tomcat by doing the following:

1. Get the latest Tomcat (currently 6.0.18) from the Apache Tomcat Website.
2. Install Tomcat by running the following:
   

   cd /opt
   tar -xzf apache-tomcat-6.0.18.tar.gz
   

3. The above will extract Tomcat under the “/opt” directory (Ex: “/opt/apache-tomcat-6.0.18″)
4. Tomcat needs these environment variables to be set:
   

   JAVA_HOME=/usr/java/jdk1.6.0_12
   CATALINA_HOME=/opt/apache-tomcat-6.0.18
   

You can set the variables above in the root user environment by default or you can set these in the Tomcat startup script. You will want to do the latter if you have several versions of Java installed and wish to use Tomcat with a specific Java version.

To adjust the root user environment, do the following:

• Under SUSE Linux 10 SP1, to set the default environment variables for root (and Tomcat), edit the “/etc/profile” file, and add the following lines to the end of the file:
  

  JAVA_HOME=/usr/java/jdk1.6.0_12
  JRE_HOME=/usr/java/jdk1.6.0_12
  CATALINA_HOME=/opt/apache-tomcat-6.0.18
  export JAVA_HOME JRE_HOME CATALINA_HOME
  

• Optionally, if you don’t want to touch “/etc/profile”, you can create an “/etc/profile.local” which should also be loaded for the root environment; if you do this, make sure to add a second “source /etc/profile.local” to your script.
• You only have to set JRE_HOME above if you need to override it because it was explicitly defined elsewhere. Otherwise, if JRE_HOME is not defined, tomcat will use JAVA_HOME instead.

To adjust the Tomcat startup script to use a specific Java version and/or use a different port than the default 8080, do the following:

• Adjust Tomcat to use the latest JDK without touching the existing environment variables.
1. Edit the “/opt/apache-tomcat-6.0.18/bin/catalina.sh” file.
2. At the top of the file, add these two lines:
   

   JAVA_HOME=/usr/java/jdk1.6.0_12
   CATALINA_HOME=/opt/apache-tomcat-6.0.18
   

3. There is another environment variable JRE_HOME which will default to JAVA_HOME if it is blank. You will want to check that your default environment doesn’t set it explicitly. If it is set explicitly, you can override by adding this line to the top of the “catalina.sh” file:
   

   JRE_HOME=/usr/java/jdk1.6.0_12
   

• If you wish to (maybe you want to run two versions of Tomcat), you can adjust Tomcat to use another port (like 7080) instead of the default 8080 by following these steps:
  1. Edit the “/opt/apache-tomcat-6.0.18/conf/server.xml” file.
  2. Replace all instances of the default port numbers with these new port numbers:
     

     Old Port	New Port
     8005	7005
     8009	7009
     8080	7080
     8443	7443

  3. While you are in this file, you might want to enable tomcat access logging by uncommenting the “AccessLog” value in server.xml to enable access logging in Tomcat. The access logs will be located in the “/opt/apache-tomcat-6.0.18/logs” directory (Ex: “localhost_access_log.2009-03-06.txt”).

Once you have installed Tomcat, you can start and stop it using these commands:

cd /opt/apache-tomcat-6.0.18/bin
sh startup.sh &
sh shutdown.sh &

Once Tomcat is started, test it out by launching your browser and doing the following:

1. Browse to http://localhost:8080/ or http://localhost:7080/ if you have changed the default port to 7080.
2. If you attempt to browse to the server from a remote machine and the server doesn’t respond, then you may need to disable the firewall.
   • Under SUSE Linux, disable the firewall by logging in as root and running “rcSuSEfirewall2 stop”. Further, to prevent the firewall from running on boot, run Yast->Firewall and change “Service Start” to Manually. If you run “chkconfig | grep firewall” afterwards, you should see that SuSEfirewall2_init and SuSEfirewall2_setup services are set to off.

Now that Tomcat is running successfully, you may wish to configure Tomcat to run on boot. One way of doing so is to configure Tomcat to be a service and registered it to be launched as part of the boot.

1. First, create the shell script file below and name it “tomcat” (or whatever service name you want to use).
   

   #!/bin/sh
   #
   # /etc/init.d/tomcat
   #
   # This is the init script for starting up the
   #  Jakarta Tomcat server
   #
   # description: Starts and stops the Tomcat daemon.
   #
   
   tomcat=/opt/apache-tomcat-6.0.18
   startup=$tomcat/bin/startup.sh
   shutdown=$tomcat/bin/shutdown.sh
   
   start() {
     echo -n $"Starting Tomcat service: "
     sh $startup
     echo $?
   }
   
   stop() {
     echo -n $"Stopping Tomcat service: "
     sh $shutdown
     echo $?
   }
   
   restart() {
     stop
     start
   }
   
   status() {
     ps -aef | grep apache-tomcat | grep -v tomcat6 | grep -v grep
   }
   
   # Handle the different input options
   case "$1" in
   start)
     start
     ;;
   stop)
     stop
     ;;
   status)
     status
     ;;
   restart)
     restart
     ;;
   *)
     echo $"Usage: $0 {start|stop|restart|status}"
     exit 1
   esac
   
   exit 0
   

2. Note that when you run a process as a service, its environment may be different from the environment you see when you log in.
   • Under SUSE Linux, you may wish to add the line below to the top of the tomcat script to ensure that the tomcat process will run with the environmental variables that you have set in “/etc/profile”.
     

     source /etc/profile
     

   • To see what environment a process is running under, you can use the following commands:
     

     ps -aef | grep tomcat
     ... you will get a process id number like 5054 ...
     cat /proc/5054/environ
     

3. Copy the file to the “/etc/init.d” directory (example: “/etc/init.d/tomcat”).
4. Give it execute permission:
   

   cd /etc/init.d
   chmod u+x tomcat
   

5. You can test the script by running this command (which will return the Tomcat process info if it is running; otherwise, it will return blank):
   

   cd /etc/init.d
   ./tomcat status
   

6. Add tomcat as a service by running the following:
   

   chkconfig --add tomcat
   

7. Now that tomcat is a service, you can issue these commands from anywhere:
   

   service tomcat start
   service tomcat stop
   service tomcat restart
   service tomcat status
   

8. Set the tomcat service to start at boot:
   

   chkconfig tomcat on
   

That is pretty much it. Your Linux server is set to automatically start Tomcat.

Now that I have my tikiwiki running on a Linux AS4 server and having felt the sting of having to recover it, I have incentive to create an automated daily backup of the mySQL database. I have decided to do the backup using a simple shell script and schedule it using the Linux crontab.

Here is the final version of the shell script (saved to "/root/sql_backup.sh"). This script will do a mySQL dump of one database to a gzip archive file in a backup directory, keep only the 10 most recent archive files (pruning the rest), and rsync the backup directory to a remote server.

#!/bin/sh
# MySQL backup script – do a backup and rsync to remote server

# mySQL
SQLROOTPASS=mysqlrootpassword
SQLDB=mysqldatabasename

# Backup Dirs
LOCALDIR=/home/backup/mysql
REMOTEHOST=myremoteserver
REMOTEDIR=/home/backup/mysql
NUMBACKUPS=10

# Create local dir
mkdir -p $LOCALDIR

# Do a mySQL backup
NOW=$(date +"%Y-%m-%d-%H-%M-%S")
echo "Doing a database backup to $LOCALDIR/$SQLDB-$NOW.sql.gz"
# Note: The following mysqldump command should be one line only!
mysqldump -u root -h localhost -p$SQLROOTPASS $SQLDB | gzip -9 > $LOCALDIR/$SQLDB-$NOW.sql.gz

# Prune old backups
FILES=$(ls -t $LOCALDIR/*.gz)
for FILE in $FILES
do
if [ $NUMBACKUPS -gt 0 ]
then
   echo Keeping $FILE
else
   echo Deleting $FILE
   rm $FILE
fi
NUMBACKUPS=$(expr $NUMBACKUPS – 1)
done

# Create remote dir
ssh $REMOTEHOST mkdir -p $REMOTEDIR

# Sync from local dir to remote dir
echo "RSyncing to $REMOTEHOST.."
rsync -rt –delete $LOCALDIR/ $REMOTEHOST:$REMOTEDIR/

Rather than explain the whole script, I’ll just point out some parts that may require more attention:

• Unfortunately, this script does contain the mySQL root password in the clear. I haven’t figured out how to do a mysqldump as a non-root SQL user yet.
• To avoid the need to include the remote server’s root password, this script will require that you have already established trust with the remote server. This avoids the need to input a password when using RSync, SSH, or SCP. See below for instructions on how to establish this trust.
• When doing an assignment to a variable in the script, don’t insert spaces after the variable name. For example, "TEMPVAR=Hello" works but "TEMPVAR =Hello" won’t.
• The format "TEMPVAR=$(…)" tells the script to execute the command inside and assign the result to the variable. This is equivalent to the format "TEMPVAR=`…`" using backward single quotes.
• In order to configure RSync to delete files on the remote system (otherwise we’ll use up all the hard drive space), we must use the -r flag (recurse into subdirectories), –delete flag, and use a directory ("$LOCALDIR/" ending with a slash) as the source. The -t flag (persist file modification time) is necessary for RSync to do delta updates.

To establish trust with a remote server, do the following:

1. Log into your server as the root user.
2. Look for this file "/root/.ssh/id_rsa.pub".
3. If the file or directory doesn’t exist, then you will need to generate the RSA public/private key:
   

   ssh-keygen -t rsa

   Just take the default inputs and your RSA keys will be generated.

4. Copy the public key "id_rsa.pub" to the remote server’s "/root/.ssh" directory.
   

   scp /root/.ssh/id_rsa.pub myremoteserver:/root/.ssh/

   If that directory does not exist, you may wish to generate the RSA public/private keys there also.

5. You will need to add the contents of "id_rsa.pub" to the end of the remote server’s "/root/.ssh/authorized_keys" file (it contains keys from other servers which the remote server should trust). If the file doesn’t exist, then create it as an empty file.
   

   cp authorized_keys authorized_keys.original
   cat authorized_keys id_rsa.pub > authorized_keys
   rm id_rsa.pub

6. You can test it out by making an SSH connection to the remote server.
   

   ssh myremoteserver

   If you are not prompted for the password, then the trust was successfully established.

Finally, let us schedule the mySQL backup to run every day at 3am using the Linux crontab:

crontab -e
0 3 * * * sh /root/sql_backup.sh > /tmp/sql_backup.output 2>&1

Some hints about using the crontab:

• "crontab -e" will allow you to edit the crontab contents.
• The crontab line format is: <minute=0-59> <hour=0-23> <day of month=1-31> <month=1-12> <day of week=0-6 (sun-sat)> <command>
• Precede the crontab line with a # (pound) character to indicate a comment which crontab will ignore.
• The "2>&1" will redirect STDERR to STDOUT, which in turn is redirected to "/tmp/sql_backup.output" above. (0=STDIN, 1=STDOUT, 2=STDERR)
• "crontab -l" will list the crontab contents.

Recently, I had to rebuild a tikiwiki server (on a Linux machine) with only a mysql database backup image. I quickly found that the backup image of a tikiwiki does not contain the version of the tikiwiki software, which is very necessary; otherwise, the database schema won’t match. Fortunately, I managed to find that the tikiwiki version was 1.8.5 from some old notes.

Here is the resulting software stack after I finished.

• Red Hat Enterprise Linux AS Release 4
• mySQL Server 4.1.20
• Apache HTTP Server 2.0.52
• PHP 4.3.9
• Tikiwiki 1.8.5 Polaris

Here’s how I went about restoring the tikiwiki (assuming that the machine already has Redhat AS4 installed):

1. Install PHP, HTTP Daemon (httpd), and supporting libraries:
   

   yum list | grep -i php
   yum install php.i386

2. Install mySQL server (mysqld) and supporting libraries:
   

   yum list | grep -i mysql
   yum install mysql-server.i386

3. Install mySQL client and the PHP-mySQL libraries:
   

   yum list | grep -i php-mysql
   yum install php-mysql.i386

4. Start both mysqld and httpd and configure them to start on reboot:
   

   chkconfig mysqld on
   service mysqld start
   chkconfig httpd on
   service httpd start

5. Configure mysqld and setup database:
   

   Set mySQL root password:
   mysqladmin -u root password ‘mypassword’
   Create a database:
   mysqladmin -u root -p create mydatabase
   Login as mySQL root user:
   mysql -u root -p
   Use main mySQL database:
   use mysql;
   Give localhost permission to access all databases:
   insert into host(host, db, Select_priv, Insert_priv, Update_priv, Delete_priv, Create_priv, Drop_priv) values(‘localhost’, ‘%’, ‘Y’, ‘Y’, ‘Y’, ‘Y’, ‘Y’, ‘Y’);
   Create a user:
   insert into user (host, user, password) values(‘localhost’, ‘myuser’, password(‘mypassword’));
   Give that user access to the created database from localhost:
   nsert into db (host, db, user, Select_priv, Insert_priv, Update_priv, Delete_priv, Create_priv, Drop_priv) values (‘localhost’, ‘mydatabase’, ‘myuser’, ‘Y’, ‘Y’, ‘Y’, ‘Y’, ‘Y’, ‘Y’);
   quit;
   Reload mySQL server:
   mysqladmin -u root -p reload

6. Reload the HTTP Server just in case:
   

   service httpd reload

7. Make sure that PHP is working by copying this testphp.php file to “/var/www/html” and browsing to “http://localhost/testphp.php”:
   

   <html>
   <head>
   <title>PHP Info Script</title>
   </head>
   <body>
   <?php
   phpinfo();
   ?>
   </body>
   </html>

8. Make sure that PHP-mySQL is working by copying this testmysql.php file to “/var/www/html”, replacing the text ”mysql_root_password” with your real password, and browsing to “http://localhost/testmysql.php”:
   

   <html>
   <title>mySQL Test Script</title>
   <body>
   <?php
   // connect and select a database
   $link = mysql_connect("localhost:3306", "root", "mysql_root_password")
   or die ("Couldn't connect: Check to make sure that:<br>" .
   "<ul><li>your MySQL server is running</li>" .
   "<li>you used the correct hostname (<tt>vergil/ovid</tt>)</li>" .
   "<li>you added a colon with your port number after the hostname</li>" .
   "<li>you used the username 'root'</li>" .
   "<li>you used the correct root password</li>" .
   "<li>you didn't forget to close a set of quotation marks</li><br><br>");
   print "Connected successfully.<br>";
   $db = "mysql";
   mysql_select_db($db) or die("Could not select the database ‘" . $db . "’. Are you sure it exists?");

   // perform an SQL query
   $query = "SELECT * FROM user";
   $result = mysql_query($query) or die("Query failed");

   // print the result of the first row (row counting starts at zero)
   printf("Host: %s<br>\n", mysql_result($result, 0, "Host"));
   printf("User: %s<br>\n", mysql_result($result, 0, "User"));
   printf("Grant privilege: %s<br>\n", mysql_result($result, 0, "Grant_priv"));

   // free result set
   mysql_free_result($result);
   // close the connection
   mysql_close($link);
   ?>
   </body>
   </html>

9. Unzip the tikiwiki 1.8.5 archive to the “/var/www/html” directory and run the setup script:
   

   chmod u+x setup.sh
   ./setup.sh
   chmod u-x setup.sh

10. Configure the tikiwiki by browsing to ”http://localhost/tiki-install.php” and inputting the tikiwiki database name, user, and password.
11. Restore the database from the backup image:
    

    mysql -u root -p
    use tikiwiki;
    source tiki_backup.sql

12. And we are done!

Some info above was provided by Linux Help – mySQL setup Guide.

If you forget your Linux root password, you can maybe reset it if you have physical access to the machine.

1. Go to the Linux server machine.
2. You will see the login prompt
3. Press Ctrl-Alt-Del (three keys at the same time)
4. The machine will reboot. If Ctrl-Alt-Del doesn’t work, try Ctrl-Alt-Backspace.
5. Wait for the machine to finish rebooting (this may take a few minutes especially if machine does a hard drive integrity check on startup).
6. You will see a lilo: prompt.
7. Enter linux single to start single user mode.
8. Enter passwd to set the new root password.
9. Enter the new root password (you will enter it twice).
10. Press Ctrl-D to start multi-user mode (which is the normal Linux mode).
11. You will see the login prompt again.
12. Enter root for username.
13. Enter your new root password.
14. You should be logged in as root.
15. Enter exit to logout.