Wednesday, December 24, 2008

Flexible Management Scripts in Groovy & JMX

It is commonplace to assert that application monitoring and management are important issues. It is less obvious though to implement such functionalities effectively. Management tasks usually require a good dose of flexibility since they deal frequently with urgent and unpredicted situations. Whereas popular implementation choices include GUIs such as web applications; or JMX with JConsole (or any other compatible client for the matter), the need for flexibility implies an implementation strategy that allows a reasonable degree of adaptation to unexpected situations and reusability. Moreover, the strategy we choose should minimize changes to the management interfaces as new requirements show up, in order to minimize the associated development and operational costs.

The strategy that I'd like to present in this post is based on Groovy and JMX. The basic idea goes as follows: On the server side we will expose our management interfaces using plain JMX; however, we will take special care in designing generic, coarse-grained management operations to promote reusability and reduce the continuous need to add new operations. On the client side, we will be scripting our management tasks using Groovy. You may already know, Groovy supports JMX nicely and intuitively via its GroovyMBean, which allows invoking JMX operations in a POJO like fashion. We will also take advantage of Groovy's concise syntax and advanced capabilities to handle the data returned by the JMX management interfaces.
It boils down to writing generic and reusable JMX interfaces, and using Groovy to write flexible scripts for management tasks, thus externalizing, as much as possible, specificities and changes from the server-side to the scripts. Scripts are cheaper to develop, modifiy and can be reused, shared or even thrown away if they are no longer useful.
Enough theorizing and let's move on to some action. Ladies and Gentlemen, let me introduce our tiny yet fully functional example application: SuperESB.

Welcome, SuperESB

SuperESB is a simple application that simulates a basic messaging system. The application accepts incoming messages by polling an input folder and delivers them to their destinations. A message consists of an originator, a destination and a payload just as in the following example:
<message>
    <originator>Ernie</originator>
    <destination>Bert</destination>
    <payload>Hello</payload>
</message>
The main component of SuperESB is MessageProcessor. Once a message is accepted, MessageProcessor attempts a delivery to the corresponding destination. A delivery is simulated by moving the message to a folder having as a name the destination of the message, under the "destinations" folder. However, if such folder does not exist the message is moved to the "undelivered" folder. We can trigger a new attempt of delivery or simply cancel a message at any time later. The interface MessageProcessor is as follows:

public interface MessageProcessor extends Runnable {

  void process(Message message);
  void deliver(Integer id);
  void cancel(Integer id);
  List<Message> getUndelivered();

}
The process method is where the handling logic of incoming messages lays. The other methods are useful to handle undelivered messages. The Message POJO appearing as a parameter of the process method is what represents a message internally. We will use this object in our management scripts so here it is:

@XmlRootElement
@XmlAccessorType(XmlAccessType.PUBLIC_MEMBER)
public class Message implements Serializable{

  public Integer id;
  public String originator;
  public String destination;
  public Date creationDate;
  public Date deliveryDate;
  public String payload;
  
  ...
}
You might have noticed, I used JAXB 2 for XML marshalling. The other thing you might need to know is that I used Spring to wire the application together. Other implementation details are not really relevant to what follows, but you can download the project and cast an eye over the code.

The Management Stuff

Now that we have the application set, we will add the management interfaces, but let's identify our needs first. Handling undelivered messages efficiently is very important. Indeed, while our system is up and running, it will keep receiving and dispatching incoming messages. The number of undelivered messages will inevitably grow up over time, due to destinations suffering downtime (simulated by the absence of their destination folder) or other processing problems. At one point, the system administrator will have to deal with all of those messages. Whether a given undelivered message is still relevant and should be delivered or cancelled depends on several factors including the originator, the destination and its age. In a real system, we can safely assert that the involved factors would be manifold and might even vary over time.

We need a method to get undelivered messages, a method to trigger a redelivery and another one to cancel a message. To keep things simple and because at this moment we might not know all the search criteria we are going to need, we will start with a single coarse-grained method with no parameters: List<Message> getUndelivered(). Here is how our first version on ManagementService looks like:

public interface ManagementService {
  
  List<Message> getUndelivered();
  void deliver(Integer id);
  void cancel(Integer id);
  
}
From an implementation perspective, we don't need something very sophisticated here. I will implement the ManagementService in a straightforward manner and expose it with org.springframework.jmx.export.MBeanExporter. The rest of the JMX configuration is in the applicationContext.xml file.

On to our first management script with Groovy. First we need to write the JMX connection code. This part is common to all of our scripts. We will create a GroovyMBean to access the remote ManagementService as a POJO:

def serviceUrl = new JMXServiceURL(
  'service:jmx:rmi://localhost/jndi/rmi://localhost:1099/myconnector')
def connector = JMXConnectorFactory.connect(serviceUrl)
def server = connector.mBeanServerConnection
println "connected to ${server.dump()}"
def managementService = new GroovyMBean(server, 'superESB:name=managementService')
Now suppose that we need to process all undelivered messages such that all messages created 1 minute prior to the current time are cancelled, whereas newer messages get delivered. Here is a Groovy script to achieve that:

def date = new Date(new Date().time - 60000)
managementService.getUndelivered().each{ message ->
  if(message.creationDate.before(date)){
    println "canceling $message.id"
    managementService.cancel(message.id)
  }
  else{
    println "deliverig $message.id"
    managementService.deliver(message.id)
  }
}
Granted, the filtering we just applied is not outstanding but in a more useful system we can do more sophisticated stuff including connection to other systems or databases. The idea here is that we used a generic management method and we relied on the script to perform specific and maybe one-time logic.

At this point, you might be concerned by performance and the fact that we had to transfer and act on every single undelivered messages individualy. To improve this, it seems only reasonable to add a batchDeliver(List<Integer> ids) management operation. In the next example script, we will use the new added operation with a slightly different filtering:

def date = new Date(new Date().time - 60000)
def messages = managementService.getUndelivered().grep{ message ->
  message.creationDate.after(date) && message.originator == 'Bert' 
    && message.destination == 'Ernie'
}
managementService.batchDeliver(messages*.id)
You got the idea. We can envisage adding other variants of getUndelivered, such as getUndeliveredSince(Date date), with one or more filtering crietria to alleviate the impact of transfering data over the wire. Finding the balance between the genericness and the number of exposed operations or parameters depends on the specificities of the system and is up to you!

Downloading and Running

You can download the application and try it out. The main class is Driver. You need to grab a copy of Spring and Groovy jars and put them on the classpath. You might need to tweak the configuration a little bit to adapt it to your environnement.