Migrating Mindtouch to new server

29 Jun

Had to move Mindtouch to a new server and had a few issues in the migration (since I ignored reading the manual).

This link helped me out in the migration process:




JMX Monitoring on Amazon EC2

19 Apr

Spent today morning figuring out how to connect JConsole or JVisualVM to a remote EC2 instance running tomcat.  These links helped.

Steps that worked for me:

  1. Download the catalina-jmx-remote.jar from the binary distribution folder of tomcat: (e.g. http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.28/bin/extras/)
  2. Copy the above jar to the lib folder of tomcat directory.
  3. Update your server.xml to include the JmxRemoteLifecycleListener
  4. Specify the -Djava.rmi.server.hostname=your.public.dns entry
  5. Make sure you do NOT specify the RMI port in your JAVA_OPTS configuration.
  6. Open both the RMI ports on your firewalls.
  7. Restart tomcat
  8. Connect using the URL: service:jmx:rmi://your.public.dns:10002/jndi/rmi://your.public.dns:10001/jmxrmi

With that, jconsole finally worked! Same is good for JVisualVM too…

Some people had an alternative approach on stack-overflow: don’t use RMI (the default); too much of a hassle; use the simpler jmxmp. (This requires the jmxremote_optional.jar to be downloaded and installed.)

P.S.: While going through the learning pangs, as a bonus, learnt how to setup a SSH tunnel using Putty. An useful arrow to have in your quiver.

ZTC Templates for PostgreSQL Zabbix Monitoring

5 Apr

If you are into Zabbix & PostgreSQL, ZTC templates are useful.  While they have code to monitor apache2, MySQL, nginx, and more, I needed just the PostgreSQL piece.  Of course, the JMX code I see in the source repository will be of interest too.  Let’s hope for a great new release…

The installation instructions work.  Check them out…

On my Ubuntu Lucid servers, I had to install the following dependencies:

aptitude install build-dep python-psycopg2 python-pip libdbd-pg-perl

After installing dependencies, I had to do this:

pip install psycopg2

The biggest gotcha with ZTC (psychopg2, rather) is the PostgreSQL authentication error. This is easily fixed by updating pg_hba.conf to use md5 authentication for local (unix sockets) instead of “ident sameuser“.

I also spent a few hours figuring out why Zabbix Server was not showing my stats even though zabbix_get was fetching results.  Turned out the query for getting database size was timing out.  Moral: don’t forget to check the zabbix server log for it has a story to tell…

mdadm assemble issue with Ubuntu Maverick / EC2

2 Apr

I have been having off and on issues with reassembling RAID 0 arrays on Ubuntu Lucid (10.04) & Maverick (10.10).  The process I follow:

  1. create new volumes from snapshots (taken using ec2-consistent-snapshot)
  2. attach them to the instance
  3. reassemble the mdadm array with mdadm -A …

It’s step 3 which sometimes starts to disassemble on Ubuntu… mdadm would repeatedly complain that device /dev/sdc1 is busy or /dev/sdc2 is busy, etc.

Every time it would report one or the other volumes in the array to be busy.  Initially I thought the volumes were corrupted or my snapshots were bad.  Turns out it is a long standing Ubuntu bug from this thread.

Here is what worked after reading that thread:

# /usr/share/mdadm/mkconf force-generate /etc/mdadm/mdadm.conf
# update-initramfs -k all -u


Reboot the system once after doing this.

Tomcat clustering with Amazon ELB

26 Mar

This one took quite some effort to fix.

Problem:  Tomcat was being run behind Amazon ELB and listening to both port 8080 and 8443. I need to redirect all traffic to https even if initiated on http by the client. So my configuration was like this:

<connector port="8080" protocol="HTTP/1.1"
redirectPort="8443" />

<connector port="8443" protocol="HTTP/1.1" sslenabled="false"
maxThreads="150" scheme="https" secure="true"
clientAuth="false" sslProtocol="TLSv1" />

However, no matter how I accessed the site, I would end up with an inoperable URL like https://proxy:8080/abc.do  While the protocol was saying ‘https’ the port was for ‘http’…

After much googling and trial and error, found a hint at this blog entry.  The trick was to add a proxyPort entry to the configuration:

<connector port="8443 proxyPort="443" protocol="http/1.1" sslenabled="false" maxThreads="150" scheme="https" secure="true"
clientAuth="false" sslProtocol="TLSv1" />

Mount GlusterFS volume at boot time in Ubuntu Lucid 10.04

26 Mar

While experimenting with GlusterFS, I had a issue with mounting of glusterfs volume at boot time.

The problem was that the fuse module was not loaded when the OS tried to mount the volume and then proceeded to hang.

The solution is quite simple: add the fuse module to /etc/modules

The /etc/fstab entries is like this:

10.x.x.x:/gluster-volume /gluster-volume glusterfs defaults 0 0

Howto: using puppet modules from Example42 on Ubuntu Lucid puppet server

15 Mar

Required reading:  Understanding Example42 modules 

Setup instructions:

1. Clone the git repository:

mkdir /usr/src/example42

cd /usr/src/example42

git clone https://github.com/example42/puppet-modules.git

2. Update the module path to include the exampl42 module by editing /etc/puppet/puppet.conf:


modulepath = /var/lib/puppet/modules:/usr/src/example42/puppet-modules

3. Restart puppetmaster for changed to take effect:

sudo /etc/init.d/puppetmaster restart

4. That’s it!  Use the classes in your node definitions!


Example (/etc/puppet/manifests/node.pp)

node ‘ptest.xxx.com’ {

   include apache