Do It Scared!

It was quiet, peaceful Tuesday night when I attempted to log into my Ubuntu Linux server as the root user. Permission denied, I got back. I tried it again. And again. Access denied. What the heck? I’m sure it was the correct password. I then tried all the passwords that it could be, from past to present. No luck. I got a sinking feeling. My root password was easy to remember and thus, easy to guess. I think my Linux server has been hacked!

No problem, I thought, I just have to reboot and log into single user mode… uh, unfortunately, my Linux server was sitting somewhere in cyberspace and I couldn’t do any of that. So I called up the service provider and they said, we didn’t touch your server and we don’t know who did. However, for $25, they can reset the root password. Having no choice, I accepted. Hours passed and finally I got an email with the new password.

I logged in, viewed the “/etc/passwd” file, and noticed two new users. Bad news, my server definitely got hacked. Good news, the hackers weren’t too sophisticated and had left a huge clue. I immediately copy the users’ info down and deleted them. Then it was off to google for what to do after a server hack. After looking at several websites, the most relevant one I found was How to restore a hacked Linux server.

What I got from the research was that once your Linux server is compromised, the best solution is to re-install the operating system. I did not want to do that because it would cost (a) money to ask the service provider to reinstall Ubuntu and (b) time for me to reinstall the applications. So I decided to play detective, find any damages, and fix them. Below are the steps I took.

1. Are there any unidentified users?

Run the following to list the users and groups:

Make a copy of the unknown users’ info, especially their usernames and ids, and delete them using the “userdel <unknown_username>” command. You should also check for and delete their home directories, located at “/home/username”.

2. Are there any strange processes running?

Run the following to list all the active processes started by the root or by the unidentified users:

Look for processes with weird names like “AkjwetSAG”; these are usually suspect. You can then run “lsof -p <process_id>” to get more info.

On my Ubuntu server, I found two processes, “avahi-daemon: chroot helper” and “/usr/sbin/console-kit-daemon –no-daemon”, that looked weird; but it turned out they were valid processes. After some quick online searches, I found that Avahi was used to detect network devices (not necessary for a server so it can be disabled or deleted) and console-kit was used for GUI related functions.

3. Are there any tracks left by the perpetrators?

Run the following to find any commands to change a password in the system:

This command returned several entries in the “auth.log” which included password changes for the unidentified user accounts. Strangely, there was no reference to a password change for the root user. By luck, there were two “Invalid user passwd from <ip_address>” messages containing the IP addresses of the perpetrators.

I then did a search on each of those two IP addresses:

I got back a ton of entries in “auth.log” which show numerous failed attempts using different usernames from these two IP addresses. I guess they kept trying until they got lucky. And they got lucky about five days ago.

Just for curiosity’s sake, I installed traceroute (it isn’t installed on Ubuntu by default) to see where the perpetrators came from:

One of the IP address came from “CHINA-NETCO.edge3.SanJose1.Level3.net” and the other from “DC-PT-01-te1-2.ip4.012.net.il”. China and Israel. But there was no guarantee that those were the origins because the perpetrators could have hacked some machines there and then used those machines to hack into my server.

4. Are there any surprises left behind?

To be on the safer side, I installed and ran two rootkit scanners recommended by the website I found:

Both reported a clean bill of health. There were one or two false positives which I investigated and cleared. Finally, some good news! It looks like the hackers didn’t do anything damaging (cross my fingers).

5. Any strange new files?

One of the recommended actions was to look for new files that were created since the break-in. But I didn’t find this useful at all; it was like looking for a needle in a haystack. (Funny, in real life, robbers take stuff. In cyberspace, they leave stuff… malicious stuff.)

On Ubuntu, I couldn’t find a way to search for newly created files since 5 days ago. So I did the next best thing, which was to search for files last modified in the past 5 days.

Unfortunately, the service provider had rebooted the machine as part of the password reset. So I got a ton of results back. Too much info to process.

6. What can I do to prepare for the next assault?

First, secure the root access. I changed the root password to an incomprehensible, random string of numbers, letters (lower and upper case), and special symbols. Try guessing that! And I will only share this root password with one other person as a backup.

Second, enable auditing of the server so we will have a record of every command issued by any user. This will make it easier to figure out what actions any user, legitimate or not, does on the server. I installed the Process Accounting (psacct or acct) service which this website recommended.

By default, the service is configured to start on bootup. To double-check, I tried using the “update-rc.d acct defaults” command but got back a strange warning “update-rc.d: warning: acct stop runlevel arguments (0 1 6) do not match LSB Default-Stop values (1)”. I didn’t know what it meant so decided to use an alternative check:

This command returned “acct on”, meaning that the Process Accounting service was configured to start on bootup.

Hopefully the above info will help you should you ever encounter the same misfortune. And maybe, the last section might help you to be better prepared.

We decided to migrate from a virtual OpenSuSE Linux server running on my desktop to a dedicated server hosting company. Unfortunately, I missed the OpenSuSE selection as an operating system choice and ended up choosing Ubuntu 64bit. Below are the initial steps I took to setup our development environment on Ubuntu.

(You can skip this section if you wish to.) For those interested, we picked CalPOP – California Point of Presence as our dedicated server hosting company for three reasons:

• They were located locally in Los Angeles, CA.
• They offered the best bang for the buck: a Dual Core Server with 4GB memory for just $69/month.
• After calling and talking to their sales and support (to ask some technical questions), I found them responsive and friendly. They seem to really care about getting our business.

Install MySQL 5

1. Run the following commands as root user:
   apt-get install mysql-server
   update-rc.d mysql defaults
   service mysql status
   mysql --version
2. “apt-get” stands for Advanced Packaging Tool and is Ubuntu’s RPM package manager.
3. The “update-rc.d” command above will configure the MySQL Server to start on bootup.

Install Apache HTTP Server

1. Run the following commands as root user:
   apt-get install apache2
   update-rc.d apache2 defaults
   service apache2 status
2. Some Apache related directories:
   • Configuration Directory: /etc/apache2
   • Debug Log Directory: /var/log/apache2
   • Default Document Root Directory: /var/www

Install PHP 5

1. Run the following commands as root user:
   apt-get install php5
   apt-get install libapache2-mod-php5
   apt-get install php5-mysql
   apt-get install php5-cli
   php -version
   service apache2 restart
2. The PHP configuration files are located under the “/etc/php5″ directory.
3. When you restart Apache, you might see this warning: “Could not reliably determine the server’s fully qualified domain name”. To resolve this warning, edit the “/etc/apache2/httpd.conf” file (strangely, it is an empty file), add a new line containing “ServerName replace_with_your_hostname”, save and restart Apache.

Install Sun Java JDK 6

1. Edit “/etc/apt/sources.list”.
2. Uncomment the two lines referring to the partner repository:
   deb http://archive.canonical.com/ubuntu lucid partner
   deb-src http://archive.canonical.com/ubuntu lucid partner
3. Run the following commands as root user:
   apt-get update
   apt-get install sun-java6-jdk
4. Double-check that the sun jdk is configured properly by running:
   java -version
   javac -version
5. If you have multiple JRE/JDKs installed, you can configured which one to use by running this command and selecting the default java installation to use:
   update-alternatives --configure java
6. Some useful directories to know:
   • Configuration Directory: /etc/java-6-sun
   • Installation Parent Directory: /usr/lib/jvm

Install Subversion

1. Run the following commands as root user:
   apt-get install subversion libapache2-svn
   svn --version
2. Create the repository directory:
   mkdir -p /var/svn/repos
   svnadmin create /var/svn/repos
   chown -R www-data:www-data /var/svn/repos
   chmod -R g+w /var/svn/repos
3. Optionally, if you are moving a subversion repository from an old server, do the following:
   (On old server)
   svnadmin dump /srv/svn/repos > backup.svn
   (Suggest that you gzip the result, copy it to the new server, and gunzip it there.)
   (On new server)
   svnadmin load /var/svn/repos < backup.svn
   chown -R www-data:www-data /var/svn/repos
   chmod -R g+w /var/svn/repos

Enable Apache HTTPS/SSL

1. Create a vhost config for SSL:
   cp /etc/apache2/sites-available/default-ssl /etc/apache2/sites-enabled/
2. Edit the “/etc/apache2/sites-enabled/default-ssl” file and modify so the SSL uses a different directory than “/var/www” and a different error log file:
   DocumentRoot /var/www-ssl
   <Directory /var/www-ssl>
   …
   ErrorLog /var/log/apache2/ssl_error.log
3. Enable the SSL module and restart apache:
   a2enmod ssl
   service apache2 restart

   The “a2enmod ssl” command just adds the “ssl.conf” and “ssl.load” files to the “/etc/apache/modes-enabled” directory so that the ssl module will be loaded by Apache on startup.

4. Browse to your server using the secured “https://localhost/” URL.

Install Subversion for Apache SSL

1. Enable the apache subversion modules:
   a2enmod dav
   a2enmod dav_svn
2. Edit the “/etc/apache2/mods-enabled/dav_svn.conf”, uncomment the lines below, and also add the “SSLRequireSSL” keyword to force only SSL access.
   <Location /repos>
     DAV svn
     SVNPath /var/svn/repos/
   
     SSLRequireSSL
   
     AuthType Basic
     AuthName "Subversion Repository"
     AuthUserFile /var/svn/dav_svn.passwd
   
     Require valid-user
   </Location>

   “Require valid-user” causes Apache to require the user to login to do any action including viewing the repository.

3. To create subversion users, use this command to generate the “var/svn/dav_svn.passwd” file:
   (For first user only)
   htpasswd -mc /var/svn/dav_svn.passwd <i>username1</i>
   (For subsequent users)
   htpasswd -m /var/svn/dav_svn.passwd <i>username2</i>

   You will be prompted to input the password for the user. Be careful to only use the “-c” create flag the first time because it will overwrite any existing password file.

4. Restart the apache server with “service apache2 restart”.
5. Browse to your subversion repository using the secured “https://localhost/repos/” URL.

Install TWiki

1. TWiki requires the Perl RCS library. Also, we want to enable the Apache Rewrite module. Run the following as the root user:
   apt-get install librcs-perl
   a2enmod rewrite
2. You can setup a secured TWiki using information from my previous post, Install TWiki on the OpenSUSE Linux Development Server.
3. Alternatively, to migrate the TWiki from the old OpenSuSE server to the new Ubuntu server, I did the following:
   (On old server, under /srv/www/htdocs-ssl)
   tar czvf twiki.tar.gz twiki
   (Copy the twiki.tar.gz to the new server.)
   (On new server, under /var/www/htdocs-ssl)
   tar xzvf twiki-files.tar.gz
   
   chown -R www-data:www-data /var/www/htdocs-ssl/twiki
   chmod -R u+rw /var/www/htdocs-ssl/twiki
   chmod -R g+rw /var/www/htdocs-ssl/twiki

   I also migrated the twiki.conf:

   • Copied it to “/etc/apache2/twiki.conf”.
   • Updated the paths in it from “/srv/www/htdocs-ssl/twiki” to “/var/www/htdocs-ssl/twiki”.
   • To allow running the TWiki configure script remotely, be sure to comment out the restriction below. When you are done, you can uncomment it back.
     #<FilesMatch "^(configure)$">
     #    SetHandler cgi-script
     #    Order Deny,Allow
     #    Deny from all
     #    Allow from localhost
     #</FilesMatch>
   • Added the following line to “/etc/apache2/sites-enabled/default-ssl”, inside the “VirtualHost” section: “Include /etc/apache2/twiki.conf”.
   • And restarted Apache with “service apache2 restart”.

   Finally, I had to run “https://localhost/twiki/bin/configure” in order to adjust the paths as follows:

   • Under “General path settings”, update {PubDir}, {TemplateDir}, {DataDir}, {LocalesDir}, and {WorkingDir}.
   • Under “Log files”, update {ConfigurationLogName}, {DebugFileName}, {WarningFileName} and {LogFileName}.

   If you encounter problems (like the TWiki not rendering your content), you can refer to the TWiki Upgrade Guide for alternative methods to migrate your TWiki. In the end, I downloaded the latest TWiki version, installed it, configured it, and then merged my “twiki/data” and “twiki/pub” content from the old to the new TWiki.

The info above is consolidated from the following sources:

• Install MySQL Server 5 on Ubuntu
• Installing PHP5 and Apache on Ubuntu
• How to install Sun Java JDK in Ubuntu 10.10 Maverick Meerkat
• Installation of Subversion on Ubuntu, with Apache, SSL, and BasicAuth

Recently, I had to setup a LAMP environment on a temporary CentOS 5.6 Linux server. (CentOS is the open source equivalent to the Red Hat Linux operating system.) Pleasantly, using Yum (Yellow dog Update, Modified), which is the CentOS and Red Hat RPM package manager, makes the task super simple. I did the following tasks while logged into a SSH shell as the root user.

Note: CentOS does not officially support the Sun Java SDK due to a licensing issue (I think). You will need to go through some hoops to get Sun Java SDK working on the CentOS. I attempted to install the latest JDK by downloading and executing the .bin, but it failed with dependency errors. I found this alternative approach, Installing Java (Sun JDK 1.6.0) on CentOS 5, but couldn’t complete it because my CentOS server started throwing segmentation faults and freezing (probably due to OS corruption or a hardware issue).

Change Hostname (Optional)

Run the command line tool “system-config-network” to change the hostname entry. This tool will update “/etc/sysconfig/network” and “/proc/sys/kernel/hostname”. You can run the “hostname” command to double-check.

Install Mysql

Run the following commands:

The last two commands will start the MySQL server and configure it to run on boot.

Install Apache

Run the following commands:

The Apache Server will have HTTPS/SSL access enabled by default. However, it will use the same htdocs directory “/var/www/html” as the HTTP. To change the HTTPS to use a different document root directory, edit the “/etc/httpd/conf.d/ssl.conf” and add the following:

To have the change take effect, restart the Apache Server by running “service httpd restart”. Check the “/var/log/httpd/ssl_error_log” for errors specific to the HTTPS service.

Install PHP

Run the following commands:

(You can combine the above packaged into one line if you wish to.)

If you need json_encode and json_decode methods, you will need to install the PHP JSON package. Unfortunately, the CentOS 5.6 Yum installs PHP 5.1 so you cannot run “yum install php-json” because it requires PHP 5.3. To install the JSON package onto PHP 5.1, run the following:

Good luck and have fun!

The info above is consolidated from:

• How to install MySql in CentOS 5.3 from shell or webmin
• CentOS – Installing Apache and PHP5

When you’re working in a small engineering team, one of the best tools for collaboration is a wiki. After looking at TikiWiki, MediaWiki, and TWiki, I decided to use TWiki. TWiki had the most sophisticated syntax for formatting a page. TWiki’s syntax trumped the downsides of using Perl and files, instead of PHP and MySQL which are used by TikiWiki and MediaWiki.

Below are notes I took when installing and securing TWiki on our OpenSUSE Linux development server.

1. Perl 5.12 comes pre-installed with OpenSUSE 11.3. Double-check that it is installed properly by running:
   perl -version
2. Browse to http://twiki.org/. Click on the “Download TWiki-5.0.1″ button to the right. Input the info requested in the download form on the right half and click on the Download button. You will download a zip archive named “TWiki-5.0.1.zip”.
3. Put the archive file onto your Linux server. As root user, unzip it to a “twiki” directory under the secure web rot directory “htdocs-ssl” and adjust the permissions so the Apache user can access it:
   unzip TWiki-5.0.1.zip -d /srv/www/htdocs-ssl/twiki
   chown -R wwwrun:www /srv/www/htdocs-ssl/twiki
4. Start configuring TWiki by creating a “LocalLib.cfg” file:
   cd /srv/www/htdocs-ssl/twiki/bin
   cp LocalLib.cfg.txt LocalLib.cfg

   Edit the resulting “LocalLib.cfg” file and update the variable “$twikiLibPath” to be the correct path:

   $twikiLibPath = "/srv/www/htdocs-ssl/twiki/lib"
5. Configure Apache by creating a “/etc/apache2/twiki.conf” file with the content below. I generated it using TWiki:TWiki.ApacheConfigGenerator and cleaned it up. (For security reasons, we are putting twiki.conf under “/etc/apache2″ instead of “/etc/apache2/conf” to prevent the TWiki Directory directives from being applicable to non-secure HTTP access.)
   # Prevent TWiki from including its own topics as URLs (used by DOS attacks).
   BrowserMatchNoCase ^$ blockAccess
   
   # ScriptAlias defines bin as a directory where CGI scripts are allowed.
   ScriptAlias /twiki/bin "/srv/www/htdocs-ssl/twiki/bin"
   
   # Alias defines pub directory as the root of file attachments.
   Alias /twiki/pub "/srv/www/htdocs-ssl/twiki/pub"
   
   # Secure file attachments by using viewfile which enforces permission
   RewriteEngine on
   RewriteCond %{REQUEST_URI} !^/+twiki/+pub/+(TWiki|Sandbox)/+.+
   RewriteRule ^/+twiki/+pub/+(.*)$  /twiki/bin/viewfile/$1 [L,PT]
   
   # Block access to typical spam related attachments
   SetEnvIf Request_URI "twiki/pub/.*\.[hH][tT][mM][lL]?$" blockAccess
   SetEnvIf Request_URI "twiki/pub/TWiki/.*\.[hH][tT][mM][lL]?$" !blockAccess
   
   # Allow everyone to run perl scripts from the bin directory.
   <Directory "/srv/www/htdocs-ssl/twiki/bin">
     AllowOverride None
     Order Allow,Deny
     Allow from all
     Deny from env=blockAccess
   
     Options ExecCGI FollowSymLinks
     SetHandler cgi-script
   
     # Password file for TWiki users
     AuthUserFile /srv/www/htdocs-ssl/twiki/data/.htpasswd
     AuthName 'Enter your login name:'
     AuthType Basic
   
     # File to return on access control error (e.g. wrong password)
     ErrorDocument 401 /twiki/bin/view/TWiki/TWikiRegistration
   
     # Limit access to configure script to localhost since it contains sensitive info.
     <FilesMatch "^(configure)$">
        SetHandler cgi-script
        Order Deny,Allow
        Deny from all
        Allow from localhost
     </FilesMatch>
   </Directory>
   
   # Allow access to the pub directory for attachments, CSS stylesheets and icons.
   <Directory "/srv/www/htdocs-ssl/twiki/pub">
     Options None
     AllowOverride None
     Order Allow,Deny
     Allow from all
     Deny from env=blockAccess
   
     # Disable execusion of PHP scripts
     php_admin_flag engine off
   
     # This line will redefine the mime type for the most common types of scripts
     AddType text/plain .shtml .php .php3 .phtml .phtm .pl .py .cgi
   </Directory>
   
   # With exception of bin and pub directories, prevent access to everything else.
   <Directory "/srv/www/htdocs-ssl/twiki/">
     Deny from all
   </Directory>
   
   # Either of this will redirect root / to twiki/bin/view
   #RewriteRule ^/$ /twiki/bin/view
   #RedirectMatch ^/$ /twiki/bin/view

   If you can’t run a browser under localhost, then make sure to comment out the “<FilesMatch “^(configure)$”>” block above so you can run the configure script remotely. If you want to redirect from the root URL to TWiki, just uncomment either “RewriteRule” or “RedirectMatch” at the end; don’t uncomment both because I’m not sure what would happen.

6. To allow access to the TWiki only from SSL/HTTPS, we will include the twiki.conf into the SSL virtual host configuration. Edit the “/etc/apache2/vhost.d/your_vhost-ssl.conf” file (which you created in this post) and add the following Include directive anywhere inside the “<VirtualHost>” block:
   <VirtualHost _default_:443>
        ...
        Include /etc/apache2/twiki.conf
        ...
   </VirtualHost>
7. The twiki.conf uses the mod_redirect module which we will need to configure Apache to load on startup. Good thing we have to restart Apache for the Twiki.conf to take effect also.
   a2enmod redirect
   rcapache2 restart
8. Run the Twiki configure script by browsing to “https://localhost/twiki/bin/configure” or equivalent. Ignore the warning about the missing “/srv/www/htdocs-ssl/twiki/lib/LocalSite.cfg” configuration file. You will create it by running the configure script this first time.
   • Input your new administrative password and click Configure button.
   • Click on “General path settings” to expand it:
     • Change the “{DefaultUrlHost}” to match your website’s domain URL.
     • Input “https://localhost” into “{PermittedRedirectHostUrls}” so you can access Twiki from a local browser.
     • Verify that the rest are correct (they should be) and click on Next.
     • Click on Save button, click on “Return to configuration” link, and enter your admin password again.
   • You will see three sections with warnings. We’ll address a couple of those warnings below, but not all. It’s okay to run TWiki with some warnings.
   • Click on “Security setup”.
     • Check “{Register}{AllowLoginName}” if you wish to allow the use of a login ID instead of WikiName.
     • I left the “{Register}{NeedVerification}” unchecked because I haven’t set up SMTP yet. This is okay because once we complete securing the TWiki, only logged-in users will be able to register new users.
     • You might want to increase the “{MinPasswordLength}” to 4 from the default 1 character.
     • You will definitely want to uncheck the “{CryptToken}{Enable}” option because it will cause a browser back action to prevent all Edits and Previews from succeeding in that session. While, I applaud the intent, this feature is flawed. When you preview a page, to re-edit it, you have to hit the Back browser button, and then if you attempt to save, CryptToken will prevent you from doing so. And once this happens, any page you attempt to edit and save will fail. (If you plan never to use Preview or use the browser back button, then you can leave this option on.)
   • Click on “Mail and Proxies”.
     • Uncheck the “{EnableEmail}” option since SMTP wouldn’t work anyhow.
     • Click Next and Save.
9. Browse to the main TWiki page at https://localhost/twiki/bin/view. You will see a nice congratulations message.
10. Click on the Registration link under “Main Web Utilities” to register a new user.
    • Just fill in the required info. Once you click Submit, your user will be created and logged into Twiki automatically.
    • Click on the TwikiUsers link on the resulting page.
    • If you want to create another user, click on the TWikiRegistration link and repeat as necessary.
    • Once you are done and back on the TWikiUsers page, log out by clicking on the “Account” menu to the top-right and selecting “Log out”.
11. For the following, you will need to log in as the administrator.
    • Go to the “Account” menu and select “Log in”. Input “admin” as the username and your administrative password.
    • Click on TWikiUsers and then TWikiGroups after “Related topics”.
    • Creating a new non-admin user group by inputting “TWikiUserGroup” into the “New Group” field and clicking on Add button. Add the WikiNames for non-administrative users to the “Set GROUP =” property. Click on the Save button.
    • Go back to TWikiGroups and click on TWikiAdminGroup to add your user as an administrator. Edit, add your WikiName to “Set GROUP =”, and Save.
    • At the bottom of the resulting page, click on TWikiAdminGroup to add your new user to the administrator group.
    • Now that we have groups and users configured, let’s restrict access to the TWiki.
      • Click on the TWikiAccessControl at the bottom.
      • Scroll down to the “Permissions settings of the webs on this TWiki site” section.
      • Click on the wrench icon before “Main” to edit the Main page permissions.
      • Edit the page and set the following to restrict access to the admin group and our user group.
        Set ALLOWWEBVIEW = %USERSWEB%.TWikiAdminGroup, %USERSWEB%.TWikiUserGroup
        Set ALLOWWEBCHANGE = %USERSWEB%.TWikiAdminGroup, %USERSWEB%.TWikiUserGroup,
                             TWikiRegistrationAgent
        Set ALLOWWEBRENAME = %USERSWEB%.TWikiAdminGroup, %USERSWEB%.TWikiUserGroup

        Note: TWikiRegistrationAgent belongs on the same line as %USERSWEB%.TWikiUserGroup.

        It is necessary to allow the TWikiRegistrationAgent to change Main.TWikiUsers to enable new user registration.

      • Save the page.
      • Go back to TWikiAccessControl (link at bottom) and repeat the above access restrictions (without TWikiRegistrationAgent) by clicking on the wrench icon before the “TWiki” link right under “Main” and changing the following:
        Set ALLOWWEBVIEW =  %USERSWEB%.TWikiAdminGroup, %USERSWEB%.TWikiUserGroup
        Set ALLOWWEBCHANGE =  %USERSWEB%.TWikiAdminGroup, %USERSWEB%.TWikiUserGroup
        Set ALLOWWEBRENAME =  %USERSWEB%.TWikiAdminGroup, %USERSWEB%.TWikiUserGroup
      • I disable the “SandBox” instead of setting permissions on it. You can do otherwise.
        Set SITEMAPLIST = off
    • Log out.
12. Browse to the main TWiki page. You will see a mandatory login form.
13. Because email was not set up, none of the users have gotten an email containing their randomly generated passwords. To fix this, do the following:
    • Edit the “/srv/www/htdocs-ssl/twiki/data/.htpasswd” file.
      • For each user line, blank out the password by deleting the random character string (encrypted password) between the first colon and second colon.
      • Save the file. Users that you have modified above will now have blank passwords.
    • Ask the users to log in with blank passwords and set their new passwords by doing the following:
      • Click on their name under the top-right Accounts menu.
      • Click on the ChangePassword link at the bottom to set their passwords.
14. That’s it. Hope that you were able to get it working.

Ease of Use Items:

• I don’t like the automatic CamelCase or WikiWord linking that TWiki has enabled by default. If I want a link, I will specify so using the square brackets. If you feel the same way, here is how you can disable this feature:
  1. Log into TWiki as a user with administrative privileges or as the default “admin” user.
  2. Click on the “Web Preferences” link on the right navigation table or at the bottom.
  3. Edit the “Web Preferences” page and add the following under “Web settings” section:
     * Set NOAUTOLINK = on
     * Set HIDE_NON_WIKI_WORD_WARNINGS = on
• The default way to go back to the parent page in TWiki is to hit the browser back button (which breaks CryptToken) or by clicking on the “Backlinks” link at the bottom (too many clicks). I love breadcrumbs and we can add a plugin to TWiki to support this function:
  1. Run the TWiki configure script (https://localhost/twiki/bin/configure). Enter your admin password.
  2. Click on “Extensions” and “Find More Extensions”. Enter your admin password.
  3. Find BreadCrumbsPlugin and click on its “Install” link to the far right. Enter your admin password.
  4. On my system, I got a weird Perl error. If you got the same, we have to install manually by doing the following:
     1. SSH into the linux server.
     2. Change directory to the TWiki installation root directory: “cd /srv/www/htdocs-ssl/twiki”
     3. There should be a script named “BreadCrumbsPlugin_installer” there. Execute it by running “Perl BreadCrumbsPlugin_installer”.
     4. Answer Yes to the re-install module prompt. The installation should complete successfully.
  5. Re-run the TWiki configure script (https://localhost/twiki/bin/configure). Enter your admin password.
  6. Go to “Plugins” and check the “{Plugins}{BreadcrumbsPlugin}{Enabled}” box. Save the change.
  7. Log into the TWiki and edit the “Web Preferences” page to add the following under “Web settings” section:
     * Set BREADCRUMBSPLUGIN_RECORDTRAIL = on
  8. In your wiki pages, add the following “%BREADCRUMBS%” variable at the top and when you view the page, it will be rendered as breadcrumb links.

Some info above derived from:

• The official TwikiInstallationGuide.
• If you decide to go with MediaWiki, check out Making MediaWiki secure (and fixing some config annoyances) for instructions on how to secure MediaWiki. I tried the instructions and they worked pretty well.

Some developers needed Java JSP and Servlet support added to our LAMP development server on OpenSUSE 11.3. To do so, we decided to use Tomcat. Follow the instructions below to install Tomcat and the Apache JK Module to integrate Apache with Tomcat.

Install Tomcat 6

1. As root user, run YaST –> Software –> Software Management. Click on the Search tab.
   • Let’s double-check that the Java SDK is installed already by inputting “jdk” into the search box and clicking on the Search button. You should see that “java-1_6_0-openjdk” and “java-1_6_0-openjdk-plugin” are already checked.
   • Input “tomcat” into the search box and click on the Search button.
   • Check “tomcat” and a bunch of the other entries will get automatically checked as dependencies.
   • Check “tomcat6-webapps” to install the tomcat webapp examples which we will use for testing below.
   • Click on the Accept button at the bottom-right to commit the changes made. YaST will exit once the installation completes.
2. Strangely, the Tomcat installation might not set some needed permissions on a few tomcat-related directories. Run the following commands to set the missing permissions:
   chmod g+w /var/log/tomcat6
   chgrp tomcat /etc/tomcat6/Catalina/localhost
   chmod g+w /etc/tomcat6/Catalina/localhost
   chmod -R g+w /var/cache/tomcat6

   Double-check that the resulting directories above have “tomcat” as group and that the group write permission is set:

   ls -l /var/log | grep tomcat6
   ls -l /etc/tomcat6/Catalina | grep localhost
   ls -l /var/cache | grep tomcat6

   If you have this issue and don’t run the above commands, you will see “Permission Denied” errors in the “/var/log/tomcat6/catalina.out” log file later.

3. Some useful info and commands for Tomcat:
   • Service Commands: rctomcat6 start/stop/restart
   • Configuration Location: /etc/tomcat6
   • Logs Location: /var/log/tomcat6
   • Webapps Location: /srv/tomcat6/webapps/
   • Workspace Cache Location: /var/cache/tomcat6
4. Configure Tomcat to start on bootup by running this command:
   insserv tomcat6
5. To test the installation, run Tomcat with “rctomcat6 start” and browse to http://localhost:8080/.

Install Apache JK Module

1. Download a pre-built Apache JK Module for OpenSUSE 11.3. Click on the “Select Mirror” link in the middle and then click on the “binary package” link at the bottom of the page.
2. Install the downloaded “apache2-mod_jk-1.2.26-6.2.x86_64.rpm” file:
   rpm -ivh apache2-mod_jk-1.2.26-6.2.x86_64.rpm

   This will install to the “/usr/share/doc/packages/apache2-mod_jk” directory and place “mod_jk.so” into Apache’s installation directory at “/usr/lib64/apache2″.

3. Create an “/etc/tomcat6/worker.properties” file for Apache to define a worker process to connect to Tomcat’s ajp13 handler port (which is enabled by default in “/etc/tomcat6/server.xml” and uses port 8009). I don’t recommend copying from the “/usr/share/doc/packages/apache2-mod_jk/worker.properties” example because it is very out-of-date. Instead, create a “/etc/tomcat6/worker.properties” file with the content below:
   # An ajp13 worker that connects to localhost:8009
   worker.list=ajp13
   
   # ajp13 worker definition
   worker.ajp13.port=8009
   worker.ajp13.host=localhost
   worker.ajp13.type=ajp13
4. Create an “/etc/apache2/conf.d/jk.conf” file to configure the Apache JK Module. I suggest copying the “/usr/share/doc/packages/apache2-mod_jk/jk.conf” example to “/etc/apache2/conf.d/jk.conf” and editing the content to match the below:
   # simple configuration for apache (for AJP connector, modul mod_jk.so)
   <IfModule mod_jk.c>
      JkShmFile /var/log/apache2/jk-runtime-status
      JkWorkersFile /etc/tomcat6/workers.properties
      JkLogFile /var/log/apache2/mod_jk.log
   
      # Log level to be used by mod_jk
      JkLogLevel error
   
      # The following line makes apache aware of the location of the /examples context
      Alias /examples "/srv/tomcat6/webapps/examples"
      <Directory "/srv/tomcat6/webapps/examples">
          Options Indexes FollowSymLinks
          allow from all
      </Directory>
   
      # The mounts all JSP files and /servlet/ uri under /examples to be handled by ajp13 tomcat
      JkMount /examples/* ajp13
   
      # The following line prohibits users from directly accessing WEB-INF
      <Location "/examples/WEB-INF/">
          deny from all
      </Location>
   </IfModule>
5. Configure Apache to load the JK module:
   a2enmod jk
   a2enmod -l

   The “a2enmod jk” command will add the “jk” module to the list of Apache’s APACHE_MODULES configuration parameter. (This configuration parameter can also be set using YaST –> System –> /etc/sysconfig Editor –> Network –> WWW –> Apache2 –> APACHE_MODULES.)

6. Restart Tomcat and Apache:
   rctomcat6 restart
   rcapache2 restart
7. To test the installation, browse to http://localhost/examples/. Try executing the servlets and JSP examples. (Strangely, I was unable to execute two JSP examples, “Basic Arithmetic” and “Functions”, without errors.)
8. If you encounter errors, check out the “/var/log/tomcat6/catalina.out”, “var/log/apache2/error_log”, and “/var/log/apache2/mod_jk.log” log files. You can increase the amount of info written to “/var/log/apache2/mod_jk.log” by editing “/etc/apache2/conf.d/jk.conf” to set “JkLogLevel info” and restarting Apache.

Some info above derived from Apache2.2 / Tomcat6 / mod_jk / Suse 11.1.

Recently, I had to setup a small development server for a team of 4-5 developers. The server needed to be accessible from the Internet and thus, needed to be secured. The server would use only free software. The server needed to provide the following development services: MySQL, Apache HTTP/HTTPS, PHP, Subversion, and SSH. For security, we will enable HTTPS/SSL on Apache and require that subversion be accessed through HTTPS.

The following are notes that I took when setting up such a server. They are not 100% comprehensive; hopefully the parts I missed are very obvious or set to acceptable defaults. I’ve indicated the specific versions of the software that I downloaded; it should be okay if you get newer versions.

Download OpenSUSE 11.3

There are many flavors of Linux out there. I just chose OpenSUSE because I am most familiar with the commercial SUSE Linux product at work. If you decide to use a different flavor such as Ubuntu or Red Hat, the Linux commands in this blog may need to be adjusted. In particular, the OpenSUSE installation program, YaST, is probably not available on other flavors.

Download the OpenSUSE distribution as an ISO image file (ex: “openSUSE-11.3-DVD-x86_64.iso”). I chose 64bit because my machine is 64bit capable. Most modern desktops and servers will support 64bit and it is recommended that you go with 64bit so your server can use more memory than 4GB.

If you are not using VMware, burn the OpenSUSE ISO image file to a DVD. Stick that DVD into your dedicated server machine and boot it up to install OpenSUSE.

Install VMware Player 3.1.3

Note: If you have a dedicated server or desktop to run OpenSUSE, you can skip this section.

Because I don’t have a dedicated machine for the Linux server, I will be using VMware Player to host it on my Windows 7 64bit machine. There are two free flavors of VMware virtual software, VMware Player and VMware Server. VMware Server is optimized for input/output (good for servers), will install as a Windows service, and thus appears to be the best choice for a Linux server. Unfortunately, from what I could find on the net, VMware Server will be end-of-life’d as of this year. So I decided to go with VMware Player.

1. Download and install the VMware Player. You will need to create a free account with VMware to do the download.
2. Run the VMware Player and click on “Create a New Virtual Machine”.
3. Select “Installer disc image file (iso)” and browse to the OpenSUSE ISO image file you downloaded above. Click Next.
4. Input a default username and password. Note that the password will also be used as the root user password. Click Next.
5. Name your virtual machine and keep the default location or change it. Click Next.
6. Take the default of 20GB or increase it. Leave the “Store virtual disk as a single file” selected. Click Next. (Note: It is very difficult to increase the size of this virtual disk later!)
7. Click on “Customize Hardware”. I recommend increasing Memory to at least 1GB. Because the server will be accessible externally, I set the Network Adapter to be “Bridged: Connected directly to the physical network”. Leave the “Replicate physical network connection state” unchecked. Click Ok.
8. Double-check the settings. Click Finish.
9. VMWare will create the virtual machine, start it, and boot from the OpenSUSE ISO image.

Install OpenSUSE 11.3

There isn’t much to say about this. The OpenSUSE installation was totally hands-free. (Kudos to the OpenSUSE development team!) The installer will reboot into OpenSUSE when the installation is complete. You should see a nice green login screen.

Use YaST to Install Apache and PHP

YaST makes finding and installing software very easy. Even better, YaST will take care of software package dependencies and will configure installed packages to work with each other. However, you do give up some control in order to use YaST. At first, I was uneasy because I didn’t know where YaST installed everything. As I got more familiar with YaST, I started liking how it embedded binaries into the OS (/usr/bin or /usr/sbin) and consolidated configuration files (/etc). And I didn’t have to edit as many configuration files as when I had installed without YaST.

1. Log into OpenSUSE as the root user.
2. Click on the green OpenSUSE icon on the bottom-left, input “yast” into the Search box, and click on the found YaST program to run it.
3. The “Software” category is pre-selected. On the right, click on “Software Management”.
4. Click on the Search tab.
   • Search for “apache”. Check “apache2″ (Apache HTTP/S Server), “apache2-mod_php5″ (Apache PHP5 Module), and “subversion-server” (Apache Subversion Module) in the search results. YaST will automatically check the must-have dependencies.
   • Search for “php” and you will noticed that YaST has pre-selected “php5″ and other items as a dependency of “apache2-mod_php5″. In addition, check “php5-mcrypt” (required for apps like PhpMyAdmin), “php5-mysql” (PHP MySQL library), and “php5-pear” (very useful for installing PHP extensions).
   • OpenSUSE pre-installs MySQL, Subversion, and SSH. To double-check, you can search for “mysql”, “subversion”, and “ssh” to see that the related “mysql-community-server”, “subversion”, and “openssh” packages are already installed (green check with black background) along with their dependencies.
5. Click on the Accept button and then the Continue button to start the installation.
6. You may prompted to insert the OpenSUSE DVD.
   • If you are using a dedicated machine and still have the OpenSUSE DVD in the DVD drive, you shouldn’t get this prompt. Otherwise, just insert the OpenSUSE DVD.
   • If you are using VMware, you will need to re-mount the OpenSUSE ISO image by clicking on the active CD icon at the botom of the VMware window, selecting Settings, selecting “Use ISO image file”, browsing to the OpenSUSE ISO image file, and clicking on OK.
   • Click on the Retry button to have YaST retry the DVD access.
7. The Software Management dialog will close once everything is installed.
8. Exit the YaST program.

Here is a quick reference for the programs installed on OpenSUSE that we are interested in:

1. MySQL
   • Service: rcmysql start/restart/stop/status (alternatively, “service mysql start/…”);
   • Binary: /usr/bin/mysql, /usr/bin/mysqld, /usr/bin/mysql_safe, …
   • Config: /etc/my.cnf, /etc/mysqlaccess.conf
   • Logs: /var/log/mysql
2. Apache HTTP/S Server
   • Service: rcapache2 start/startssl/restart/stop/status (alternatively, “service apache2 start/…”)
   • Binary: /usr/sbin/httpd2-prefork
   • Config: /etc/apache2/httpd.conf, /etc/sysconfig/apache2
   • Logs: /var/log/apache
   • Document Root: /srv/www/htdocs
3. PHP
   • Binary: /usr/bin/php
   • Config: /etc/php5/apache, /etc/php5/cli
   • Apache Config: /etc/apache2/conf.d/php5.conf
4. Subversion
   • Binary: /usr/bin/svn
   • Apache Config: /etc/apache2/conf.d/subversion.conf
   • Note: /usr/bin/svnserve is a small, standalone Subversion server. There is no need to run svnserve as subversion access will be provided through Apache.
5. SSH
   • Service: rcsshd start/restart/stop (alternatively, “service sshd start/…”)
   • Binary: /usr/bin/ssh, usr/sbin/sshd
   • Config: /etc/ssh_config, /etc/sshd_config

Test the Installation (HTTP and PHP)

Let’s run some test to ensure that everything is installed correctly. Log in as root, run Terminal, and input the following commands:

• Test MySQL:
  rcmysql status
  rcmysql start
  mysql -u root
  mysql> show databases;
  mysql> use mysql;
  mysql> show tables;
  mysql> quit
• Test Apache:
  rcapache2 status
  rcapache2 start
• Create index.html and phpinfo.php:
  echo "<html><body><h3>Hello</h3></body></html>" > /srv/www/htdocs/index.html
  cat > /srv/www/htdocs/phpinfo.php << EOL
  > <?php phpinfo(); ?>
  > EOL
• Browse to http://localhost/ to test HTTP and http://host/phpinfo.php to test PHP.

Configure Subversion on Apache

1. Install apache modules to enable subversion (svn) by running these commands as the root user:
   a2enmod dav
   a2enmod dav_svn
   a2enmod -l

   The last command “a2enmod -l” will list the configured apache modules to run and you should see “dav” and “dav_svn” listed. Later on, after you restart Apache, you can run “httpd2 -M” to see the actual modules loaded by Apache.

2. Edit “/etc/apache2/conf.d/subversion.conf” and add the following right after “<IfModule mod_dav_svn.c>” and before “</IfModule>”:
   <Location /repos>
     DAV svn
     SVNPath /srv/svn/repos/
   
     # Limit write permission to list of valid users.
     <LimitExcept GET PROPFIND OPTIONS REPORT>
       # Require SSL connection for password protection.
       # SSLRequireSSL
   
       AuthType Basic
       AuthName "Authorization Realm"
       # path to password file
       AuthUserFile /srv/svn/user_access/passwd
       Require valid-user
     </LimitExcept>
   </Location>

   (Note: Use SVNParentPath instead of SVNPath if you want to have multiple repositories under “/srv/svn/repos/” without having to modify the subversion.conf file for each.)

3. Create the repository at “/srv/svn/repos” directory to match what is in subversion.conf:
   mkdir -p /srv/svn/repos
   svnadmin create /srv/svn/repos
   chown -R wwwrun:www /srv/svn/repos

   The reason to run the last “chown” command is so that the Apache user (“wwwrun”) can access the repository directory and its contents. Otherwise, you will get permission denied errors when running Subversion commands against the repository.

4. Create a Subversion password file “/srv/svn/user_access/passwd” (to match what is in subversion.conf) and add the first user:
   mkdir -p /srv/svn/user_access
   htpasswd2 -mc /srv/svn/user_access/passwd <username>

   Replace “<username>” with the username of your first Subversion user. You will be prompted to input a password for that user. To create additional users at any time, run this command:

   htpasswd2 -m /srv/svn/user_access/passwd <username2>

   Note: The “htpasswd2″ command for adding subsequent users does not include the “-c” flag. Be careful because the “-c” flag will re-create the password file and you will lose all pre-existing Subversion users!

5. Restart Apache so that all the changes will take effect.
   rcapache2 restart
6. Test Subversion by browsing to http://localhost/repos/.
7. You can import an existing project directory into Subversion and then check it back out by running the following commands:
   svn import ./project http://localhost/repos/project --username <username>
   svn co http://localhost/repos/project ./project_test

   You will need to specify your “<username>” in the first Subversion command. Once you do so, Subversion will remember the username and you will not need to include it in subsequent Subversion commands.

Configure SSL on Apache

To get SSL (aka HTTPS) working on the Apache HTTP/S Server, we will need to install a certificate for our server. Instead of paying for an official certificate from companies like VeriSign, I decided to generate a self-signed root CA (Certifcate Authority) certifcate and server certificate (signed by that root CA).

1. Create the certificates using the provided OpenSUSE “mkcert.sh” script. Note that you must run the “mkcert.sh” script from its directory in order for everything to work properly.
   cd /usr/share/doc/packages/apache2
   ./mkcert.sh make --no-print-directory /usr/bin/openssl /usr/sbin/ custom

   The script will prompt you with a bunch of selections:

   • Note: If you want to leave a field blank below, just input “.”. Unless otherwise stated, just take the default selection.
   • For section “STEP 2: Generating X.509 certificate signing request for CA [ca.csr]“:
     • Enter “US” (United States) for “Country Name”.
     • Enter the full state name for “State or Province Name” (ex: “California”).
     • Enter the full city name for “Locality Name” (ex: “Los Angeles”).
     • Enter the full company name for “Organization Name” (ex: “Snake Oil Company”).
     • Take the default “Certificate Authority” for “Organization Unit Name”.
     • Enter the full company name with “CA” (for Certificate Authority) at the end for “Common Name” (ex: “Snake Oil Company CA”).
     • Enter an email address (ex: “admin@snakeoil.com”).
   • For section “STEP 5: Generating X.509 certificate signing request for SERVER [server.csr]“:
     • Except for “Common Name”, repeat the inputs from STEP 2 for “Country Name”, “Start or Province Name”, etc.,
     • For “Common Name”, input the fully-qualified domain name for your server (ex: “www.snakeoil.com”).
   • For section “STEP 7: Enrypting RSA private key of CA with a pass phrase for security [ca.key]“:
     • Answer Y for yes and input a password.
     • Note: We want a password because we will be distributing the root CA certificate so users can manually install it into the browser as a trusted root CA certificate (which is used to automatically trust websites signed by it).
   • For section “STEP 8: Enrypting RSA private key of SERVER with a pass phrase for security [server.key]“:
     • Answer N to avoid inputting a password.
     • Note: If you input a password for the server key, you will need to input this password every time you start Apache. There is really no need for a password because we will not be distributing this key.
   • You should get a success message. Ignore the text about where the certificates are stored. The certificates are actually stored in the following locations where Apache can make use of them:
     /etc/apache2/ssl.key/ca.key (CA RSA Private Key 1024bit; encrypted with password)
     /etc/apache2/ssl.crt/ca.crt (CA X.509 Certificate Signed by itself)        
     /etc/apache2/ssl.key/server.key (Server RSA Private key 1024bit, not encrypted with password)
     /etc/apache2/ssl.crt/server.crt (Server X.509 Certifcate Signed by CA)

     (These certificates are valid for one year.)

2. Copy the root CA certificate (“ca.crt”) to the non-SSL document root so that users can download and add it as a trusted root CA certifcate to their browser.
   cp /etc/apache2/ssl.crt/ca.crt /srv/www/htdocs/ca.crt
3. We need to create a virtual host for the HTTPS port 443:
   • Make a copy of “/etc/apache2/vhosts.d/vhost-ssl.template” in the same directory and name it something like “your_vhost-ssl.conf” (ex: “snakeoil-ssl.conf”). You can name the file to whatever you want; you just need to keep the “.conf” extension.
   • Edit “/etc/apache2/vhosts.d/your_vhost-ssl.conf” to make the following changes.
   • Uncomment the “ServerName” and “ServerAdmin” entries. Please replace the “www.snakeoil.com” and “admin@snakeoil.com” entries below with your server’s hostname (this should match what you inputted as the “Common Name” in the server certificate) and email address.
     DocumentRoot "/srv/www/htdocs-ssl"
     ServerName www.yourhostname.com
     ServerAdmin admin@yourhostname.com
     ErrorLog /var/log/apache2/ssl_error_log
     TransferLog /var/log/apache2/ssl_access_log
   • Uncomment “#SSLCertificateChainFile /etc/apache2/ssl.crt/ca.crt” by removing the “#” pound character. Add the following line: “SSLCACertificateFile /etc/apache2/ssl.crt/ca.crt”. Also, make sure “SSLCertificateFile” and “SSLCertificateKeyFile” are set. You should end up with the following:
     SSLCertificateChainFile /etc/apache2/ssl.crt/ca.crt
     SSLCACertificateFile /etc/apache2/ssl.crt/ca.crt
     SSLCertificateFile /etc/apache2/ssl.crt/server.crt
     SSLCertificateKeyFile /etc/apache2/ssl.key/server.key

     The first line will add our self-signed root CA certifcate into the server certificate’s authorization chain. The second line will identify our self-signed root CA certificate as the server certificate’s root CA. The third and four line identifies the server certificate that Apache HTTPS should use.

   • Because we are not re-using the “/srv/www/htdocs” directory as a parent for our vhost directory, we need to define the access options for the “/src/www/htdocs-ssl” directory. Add the following near the bottom of the “<VirtualHost>” definition before “</VirtualHost>”. Look for the existing “<Directory “/src/www/cgi-bin”> … </Directory>” section and add the following right after it:
     <Directory "/srv/www/htdocs-ssl">
       Options None
       AllowOverride None
       Order allow,deny
       Allow from all
     </Directory>
4. The apache “ssl” module should already be installed. Run the “a2enmod -l” command and check that “ssl” is listed.
5. Create the vhost document root directory and add some test files to it:
   mkdir -p /srv/www/htdocs-ssl
   echo "<html><body><h3>Encrypted</h3></body></html>" > /srv/www/htdocs-ssl/index.html
   cp /srv/www/htdocs/phpinfo.php /srv/www/htdocs-ssl/phpinfo.php
6. Restart Apache to have the SSL changes take effect. Unfortunately, if you restart using “rcapache2 restart”, SSL won’t work!
   • The problem is that to enable SSL, you need to run Apache with the SSL flag set. The way to do that is to stop Apache with “rcapache2 stop” and then start Apache with the special “rcapache2 startssl” command.
   • To configure “rcapache2 start/restart” to run with the SSL flag (like with “rcapache2 startssl”), run the following command:
     a2enflag SSL

     This will modify the “/etc/sysconfig/apache2″ file to add “SSL” to the “APACHE_SERVER_FLAGS” variable to be:

     APACHE_SERVER_FLAGS="SSL"

     After you make this change, just run “rcapache2 restart” like normal. (Alternatively, you can run YaST to set “SSL” in System –> /etc/sysconfig Editor –> Network –> WWW –> Apache2 –> APACHE_SERVER_FLAGS.)

7. Test the Apache SSL support:
   • Browse to “http://www.yourhostname.com/ca.crt” (replace “yourhostname” with your server’s hostname) to download the root CA certificate. On Firefox, select the “Trust this CA to identify web sites” option and click the OK button.
   • Browse to “https://www.yourhostname.com/” to make sure HTML works under SSL.
   • Browse to “https://www.yourhostname.com/phpinfo.php” to make sure PHP works under SSL.
   • Browse to “https://www.yourhostname.com/repos/” to make sure Subversion works under SSL.
8. If you run into trouble, here are some troubleshooting tools:
   • To view the open ports, run “netstat -vatn”. You should see port 443 (the default HTTPS port) listed.
   • To view the virtual host configuration, run “httpd2 -D SSL -S”.
   • To test using an SSL client, run “openssl s_client -connect localhost:443″.

Secure Subversion

Now that we know subversion works under SSL, let’s secure Subversion so it will only work under HTTPS/SSL. This is very simple to do. Edit “/etc/apache2/conf.d/subversion.conf” and uncomment the “#SSLRequireSSL” option. In addition, comment out or delete the “<LimitExcept>” and “</LimitExcept>” lines (just the two lines, not the content between them) to force the Subversion user to provide the password for all actions including viewing the repository. The result should look like:

Configure SSH

Surprise! You don’t need to configure anything to get SSH working. You just need to start the SSH server with this command.

Configure MySQL, Apache, and SSH to Start On Bootup

To configure MySQL and Apache to start at bootup, log in as root, run Terminal, and input the following commands:

Alternatively, run YaST, go to System, and run “System Services (Runlevel)”. Select “apache2″ and click on the Enable button. Do the same for “mysql” and “sshd”.

Miscellaneous

If Firefox on OpenSUSE can’t browse to certain sites, the issue might be caused by IPv6 being enabled by default. To disable IPv6:

1. Browse to “about:config”.
2. Input “IPv6″ into the “filter” to search.
3. Click on “network.dns.disableIPv6″ once to enable; the Value should turn to “true”.

If the DNS resolution doesn’t seem work in every situation, you may need to manually configure the DNS servers. For example, you can “ping google.com” but you can’t run “telnet google.com 80″ (or SSH and scp) without getting a “Could not resolve hostname” error. To set the DNS servers to use Google’s DNS servers:

1. Run YaST –> Network Devices –> Network Settings –> Hostname/DNS tab.
2. Input 8.8.8.8 into the “Name Server1″ field and 8.8.4.4 into the “Name Server2″ field.
3. Click Ok to apply the changes.

To secure the MySQL database, you can run the provided “/usr/bin/mysql_secure_installation” script (I haven’t tried this yet). This script will perform tasks like setting the MySQL root password and disabling the remote root login.

Some info above derived from the following websites:

• Svn Server?
• OpenSUSE 11.1 Reference – Setting Up a Secure Web Server with SSL
• host names cannot be resolved except for ping

Recently, I automated the daily build and deployment of a web application to a tomcat server. My automated build made calls to stop and restart the tomcat service using the tomcat service script.

Unfortunately, I found that tomcat sometimes will hang when quitting. As a result, after several days, I found several instances of tomcat running and the web server not serving pages. The cause was that the first tomcat instance, which owned the http port, was hung and the subsequent instances could not grab the http port.

The fix was to manually kill all the tomcat instances and then start up a new instance. Because I did not wish to potentially do this manually every day, I decided to beef up the tomcat service script to check for and kill hung tomcat instances.

Below is the updated “/etc/init.d/tomcat” script:

That is as bulletproof as I could make it on a first pass. Hopefully someone will find this useful.

There is no official subversion binary distribution for SUSE Linux. However, it is pretty easy to compile subversion on your SUSE Linux server.

1. If you have an older version of subversion which you wish to remove before installing the latest, do the following cleanup:
   • Run Yast2, go to Software->Software Management, search on “svn”, and remove it.
   • Manually delete any remnant subversion files by doing the following:
     rm -f /usr/local/lib/libsvn*
     rm -f /usr/local/lib/libapr*
     rm -f /usr/local/lib/libexpat*
     rm -f /usr/local/lib/libneon*
2. Make sure that you have “autoconf” installed by running “rpm -qa | grep autoconf”. If you do not have autoconf installed, run Yast2, go to Software->Software Management, and search for “autoconf” to install it.
3. Get the following source distributions or newer from subversion website: “subversion-1.6.2.tar.gz” and “subversion-deps-1.6.2.tar.gz”.
4. Extract both archives into the same directory on your SUSE Linux server:
   gunzip *.gz
   tar -xvf subversion-1.6.2.tar
   tar -xvf subversion-deps-1.6.2.tar
   chown -R root:root subversion-1.6.2
   mv subversion-1.6.2 /opt

   You will end up with the subversion binaries in the “/opt/subversion-1.6.2″ directory.

5. Compile and install subversion with these commands:
   cd /opt/subversion-1.6.2
   sh ./autogen.sh
   ./configure --with-ssl
   make install

   You may see some errors when running “autogen.sh” or “configure”… just ignore them and hope that “make install” will work. (Some websites recommend using “checkinstall” instead of “make install” but my SUSE Linux server did not recognized “checkinstall”. “checkinstall” is supposed to generate an rpm package and update the rpm database appropriately to allow for easy uninstalls.)

6. Double-check by running “man svn”. Then attempt to checkout a project with a command like:
   svn --username bob --password vila co http:/serverhost/repos/myproject

Security Warning: The subversion password will be stored in clear text in a file “~/.subversion/auth/svn.simple”. If you are on a shared system, you might wish to restrict the directory permissions to prevent others from viewing that file.

Whew! Hopefully that wasn’t so bad to follow.

When using Yast2 to install or modify my SUSE Linux server, it will prompt me to insert the original installation CDs. This is bothersome if my SUSE Linux server is not easy to access physically. To avoid having to deal with CDs, we can copy the contents of the installation CDs to the hard drive.

Copy the contents of the installation CDs to the hard drive:

1. Create a mount point “/dvd” on your hard drive and a directory to place the CDs (example: “/opt/suse10sp1″).
   mkdir /dvd
   mkdir /opt/suse10sp1
2. Insert each CD and run the following commands to mount the CD, copy the CD contents to “/opt/suse10sp1″, and eject the CD:
   mount /dev/dvd /dvd
   cp -r /dvd /opt/suse10sp1
   mv /opt/suse10sp1/dvd /opt/suse10sp1/cd1
   umount /dvd
   eject /dev/dvd

   For each CD, you can increment “/opt/suse10sp1/cd1″ (example: /cd2, /cd3, and /cd4). If “mount /dev/dvd /dvd” does not work, try “mount /dev/hda /dvd” (“/dev/dvd” is just a soft link to “/dev/hda”).

Configure Yast2 to look for installation files on the hard drive:

1. Run Yast2
2. Go to Software -> Installation Source
3. You should see a source entry for the CD drive that looks like “cd:///?devices=/dev/hda”. Select it and hit the Delete button.
4. Click on the Add button, select Local Directory, and browse to the path “/opt/suse10sp1/cd1″. (Once you give Yast2 the first CD location, it will figure out the other CDs’ locations based upon the first.)
5. The local path should show up as a source entry that looks like “dir:///opt/suse10sp1/cd1″.
6. Select our new source entry and unselect the “Synchronize Changes with ZENworks” (per source stting) as it is no longer applicable.

And we are done. Yast2 should no longer prompt you to insert the installation CDs.

Apache Tomcat is a Java Servlet container which can be run as a standalone HTTP server. The instructions below will walk you through installing and getting Tomcat working on your Linux box. I’m doing this on a SuSE Linux server but it should be applicable to other Linux flavors like Red Hat.

Note: All the instructions below are executed using the root user account.

First, your Linux server must have Java installed because Tomcat requires it. If you don’t have Java installed already or want the latest, do the following:

1. Get the latest JDK (currently JDK 6 update 12) from the Sun Java Website.
2. Download the jdk bin file, give it execute permission, and run it. It will extract an RPM file.
3. Install the JDK RPM by running this command:
   rpm -ivh jdk-6u10-linux-i586.rpm
4. The JDK will be installed under the “/usr/java” directory (Ex: “/usr/java/jdk1.6.0_12″).

Now, you can install the latest Tomcat by doing the following:

1. Get the latest Tomcat (currently 6.0.18) from the Apache Tomcat Website.
2. Install Tomcat by running the following:
   cd /opt
   tar -xzf apache-tomcat-6.0.18.tar.gz
3. The above will extract Tomcat under the “/opt” directory (Ex: “/opt/apache-tomcat-6.0.18″)
4. Tomcat needs these environment variables to be set:
   JAVA_HOME=/usr/java/jdk1.6.0_12
   CATALINA_HOME=/opt/apache-tomcat-6.0.18

You can set the variables above in the root user environment by default or you can set these in the Tomcat startup script. You will want to do the latter if you have several versions of Java installed and wish to use Tomcat with a specific Java version.

To adjust the root user environment, do the following:

• Under SUSE Linux 10 SP1, to set the default environment variables for root (and Tomcat), edit the “/etc/profile” file, and add the following lines to the end of the file:
  JAVA_HOME=/usr/java/jdk1.6.0_12
  JRE_HOME=/usr/java/jdk1.6.0_12
  CATALINA_HOME=/opt/apache-tomcat-6.0.18
  export JAVA_HOME JRE_HOME CATALINA_HOME
• Optionally, if you don’t want to touch “/etc/profile”, you can create an “/etc/profile.local” which should also be loaded for the root environment; if you do this, make sure to add a second “source /etc/profile.local” to your script.
• You only have to set JRE_HOME above if you need to override it because it was explicitly defined elsewhere. Otherwise, if JRE_HOME is not defined, tomcat will use JAVA_HOME instead.

To adjust the Tomcat startup script to use a specific Java version and/or use a different port than the default 8080, do the following:

• Adjust Tomcat to use the latest JDK without touching the existing environment variables.
1. Edit the “/opt/apache-tomcat-6.0.18/bin/catalina.sh” file.
2. At the top of the file, add these two lines:
   JAVA_HOME=/usr/java/jdk1.6.0_12
   CATALINA_HOME=/opt/apache-tomcat-6.0.18
3. There is another environment variable JRE_HOME which will default to JAVA_HOME if it is blank. You will want to check that your default environment doesn’t set it explicitly. If it is set explicitly, you can override by adding this line to the top of the “catalina.sh” file:
   JRE_HOME=/usr/java/jdk1.6.0_12
• If you wish to (maybe you want to run two versions of Tomcat), you can adjust Tomcat to use another port (like 7080) instead of the default 8080 by following these steps:
  1. Edit the “/opt/apache-tomcat-6.0.18/conf/server.xml” file.
  2. Replace all instances of the default port numbers with these new port numbers:
     

     Old Port	New Port
     8005	7005
     8009	7009
     8080	7080
     8443	7443

  3. While you are in this file, you might want to enable tomcat access logging by uncommenting the “AccessLog” value in server.xml to enable access logging in Tomcat. The access logs will be located in the “/opt/apache-tomcat-6.0.18/logs” directory (Ex: “localhost_access_log.2009-03-06.txt”).

Once you have installed Tomcat, you can start and stop it using these commands:

Once Tomcat is started, test it out by launching your browser and doing the following:

1. Browse to http://localhost:8080/ or http://localhost:7080/ if you have changed the default port to 7080.
2. If you attempt to browse to the server from a remote machine and the server doesn’t respond, then you may need to disable the firewall.
   • Under SUSE Linux, disable the firewall by logging in as root and running “rcSuSEfirewall2 stop”. Further, to prevent the firewall from running on boot, run Yast->Firewall and change “Service Start” to Manually. If you run “chkconfig | grep firewall” afterwards, you should see that SuSEfirewall2_init and SuSEfirewall2_setup services are set to off.

Now that Tomcat is running successfully, you may wish to configure Tomcat to run on boot. One way of doing so is to configure Tomcat to be a service and registered it to be launched as part of the boot.

1. First, create the shell script file below and name it “tomcat” (or whatever service name you want to use).
   #!/bin/sh
   #
   # /etc/init.d/tomcat
   #
   # This is the init script for starting up the
   #  Jakarta Tomcat server
   #
   # description: Starts and stops the Tomcat daemon.
   #
   
   tomcat=/opt/apache-tomcat-6.0.18
   startup=$tomcat/bin/startup.sh
   shutdown=$tomcat/bin/shutdown.sh
   
   start() {
     echo -n $"Starting Tomcat service: "
     sh $startup
     echo $?
   }
   
   stop() {
     echo -n $"Stopping Tomcat service: "
     sh $shutdown
     echo $?
   }
   
   restart() {
     stop
     start
   }
   
   status() {
     ps -aef | grep apache-tomcat | grep -v tomcat6 | grep -v grep
   }
   
   # Handle the different input options
   case "$1" in
   start)
     start
     ;;
   stop)
     stop
     ;;
   status)
     status
     ;;
   restart)
     restart
     ;;
   *)
     echo $"Usage: $0 {start|stop|restart|status}"
     exit 1
   esac
   
   exit 0
2. Note that when you run a process as a service, its environment may be different from the environment you see when you log in.
   • Under SUSE Linux, you may wish to add the line below to the top of the tomcat script to ensure that the tomcat process will run with the environmental variables that you have set in “/etc/profile”.
     source /etc/profile
   • To see what environment a process is running under, you can use the following commands:
     ps -aef | grep tomcat
     ... you will get a process id number like 5054 ...
     cat /proc/5054/environ
3. Copy the file to the “/etc/init.d” directory (example: “/etc/init.d/tomcat”).
4. Give it execute permission:
   cd /etc/init.d
   chmod u+x tomcat
5. You can test the script by running this command (which will return the Tomcat process info if it is running; otherwise, it will return blank):
   cd /etc/init.d
   ./tomcat status
6. Add tomcat as a service by running the following:
   chkconfig --add tomcat
7. Now that tomcat is a service, you can issue these commands from anywhere:
   service tomcat start
   service tomcat stop
   service tomcat restart
   service tomcat status
8. Set the tomcat service to start at boot:
   chkconfig tomcat on

That is pretty much it. Your Linux server is set to automatically start Tomcat.