A Groovy Way to Write SAS XPT

Recently our development team was posed with somewhat of a challenge. Besides having various requests from our sponsors, our data management group had a desire to obtain SAS XPT clinical extracts from our EDC system without requiring SAS tools (and expensive SAS licenses) to do so.

First off, why SAS XPT?  SAS XPT is the format that the FDA requires for New Drug Applications as part of the drug development process. It's the least common denominator for representing and transferring clinical data.  And, because of US law (and the fact that it's just the right thing to do), the FDA must remain vendor neutral.  As part of that neutrality, SAS XPT must be an open format.  From the SAS FDA FAQ:

Q.	What do you mean, the XPORT format is "open?"
A.	Specifications for the XPORT transport format are in the public domain. Data can be translated to and from the XPORT transport format to other commonly used formats without the use of programs from SAS Institute or any specific vendor

The specification can be found here.  My initial thinking was, there's gotta be some sort of open source implementation with this stuff; and apparently I wasn't the only one thinking that.  I had seen some things with R where one could read SAS XPT, but nothing to write it.  Why not?  I'm guessing there isn't much out there because it's hard to do, and once people crack it, they don't want to share :)

In my opinion, the hardest part of the specification is the fact that, "All integers are stored using IBM-style integer format, and all floating-point numbers are stored using the IBM-style double."  Really, instead of using the common IEEE format, the FDA mandates something as archaic and ancient as IBM Floating Point?  Fortunately, one of my developers is good with C programming, and I had just enough experience on the mainframe to come up with a solution.

So yeah... our reporting engine runs on a JVM, and Groovy was a natural fit to develop this with.  Why Groovy instead of just Java?

• Groovy I/O - leveraging the dynamic left shift operator is just a natural way to write to an java.io.OutputStream, and build the XPT file.  Furthermore, with Groovy's management of Exceptions, I don't have to constantly handle java.io.IOException.
• Duck Typing - as we write out each data point, if the object acts like a duck, quacks like a duck... it's a duck.  With SAS XPT, you define a type for each column.  And since Groovy has a dynamic toDouble() method on java.lang.String and all implementations of java.lang.Number, the code doesn't care about the underlying object type passed in... it's just converted to a double.
• Because Groovy is still Java - even using the Groovy programming language, we can still do the bit banging necessary to convert IEEE to IBM's Floating Point.
• Incredibly useful dynamic methods for implementing some of the subtleties of the specification.  A few gems are java.lang.String's pad methods, and java.util.Date's getAt method.

Obviously all of this is possible in just Java (or other programming languages, for that matter), but it's just better using Groovy :)

World, meet iQ. iQ, meet world.

This week in Chicago we formally introduced the world to the iQ - a truly disruptive technology that will revolutionize the way drugs are developed and tested for cardiac safety.

I've talked about this project in the past from a technical perspective, but for non-geeks: at a high level this thing makes it dead simple for pharmaceutical companies to ensure that ALL of their later-phase drug trials have board certified cardiologists looking things over.

Here's a picture from the DIA trade show. Over three days, we demonstrated nearly 60 ECG acquisitions. In each case we were able to correctly identify Andrew (the poor guy with no shirt) from our subject database using voice biometrics 100% of the time.

Best. Groovy. Method. Ever.

Well, maybe not the best method ever... but it's gotta be damn close :) In just a few lines of code I'm doing a lot of really Groovy stuff!

I was looking for a quick and dirty way of converting JSON into some simple groovy objects that I have. The method recursively walks the object hierarchy and then populates corresponding values that are present in the JSON. Enjoy!

  /**
   * Slick way of using reflection to populate value 
   * objects with JSON.  Performs a DEEP copy in order 
   * to populate all of the dependent pogo properties
   *
   * @param obj object to populate - pogo
   * @param json grails.converters.deep.JSON
   * @return the obj param
   */
  protected def populateFromJson(def obj, def json) {
    obj.metaClass.properties.each {metaBeanProperty ->
      if (metaBeanProperty.modifiers == Modifier.PUBLIC) {
        def val = null
        String jsonVal = json[metaBeanProperty.name]
        if (metaBeanProperty.type == String) {
          val = jsonVal
        } else if (Number.isAssignableFrom(metaBeanProperty.type)) {
          if (jsonVal?.isNumber()) val = jsonVal.invokeMethod("to${metaBeanProperty.type.simpleName}", null)
        } else if (metaBeanProperty.type == Date) {
          if (jsonVal) val = Date.parse("yyyy-MM-dd'T'HH:mm:ss'Z'", jsonVal)
        } else if (metaBeanProperty.type == Boolean) {
          if (jsonVal) val = jsonVal.toBoolean()
        } else if (metaBeanProperty.type.name.startsWith("com.mycomp")) { 
          val = populateFromJson(metaBeanProperty.type.newInstance(), json[metaBeanProperty.name])
        }
        obj[metaBeanProperty.name] = val
      }
    }
    return obj
  }

Inter JVM Communication

Ever have an application that should only have one instance of it running?  Well, I do... I have a Swing application, and it integrates with a device via USB.  I want to ensure that there's only one player in the game at a given time.

How can one Java process know if another with the same application is running?  There seems to be a lot of suggestions out there: RMI, Cajo, Terracotta, etc, but since I am using Groovy, I figured there'd be an easier way.  Turns out (for my use case), there is.  Some of the dynamic I/O methods added to java.net.Socket and java.net.ServerSocket make this a snap.

My basic strategy goes something like this:

• Implement a single method to determine if the process can run.  If it can, fire up a server process, and then store and encrypt the server's port number in a File and return true.  If it can't, return false.
• On invocation, if there is no file, or an invalid port number (non-Integer), it's safe to say that no other Java process for the given application is running.
• If there is a valid port, create a socket connection and issue a simple 'ping' following an incredibly simple protocol.  If the server process is not available, or if that process doesn't respond properly to the ping, it's safe to say the application in question is not running.
• If, however, the ping returns as expected, we know that this application is running already, and we want to return false and then shut down.
• If it's determined that no other process is running, start up a server socket in another thread, encrypt the port created, and store it in the port file.

The obvious weakness in this strategy is that a user could potentially modify the encrypted file, but for my use case, we can live with that risk.

Also, this solution is certainly doable with just Java, but it's just a lot damn easier with some of the slick dynamic Groovy I/O methods.  Here's the code:

  /**
   * By examining the file, determine if a a socket thread is holding on to the specified port.  If we can 
   * communicate on that port, it's safe to say that another instance of this app is running, and this 
   * method returns false
   *
   * @param portFile file that contains encrypted port.  If null, or invalid, this method will assume 
   *        we can run
   * @return true if this app is launchable
   */
  static boolean isLaunchable(File portFile) {
    boolean ret = true
    if (portFile.exists()) {
      Socket client = null
      try {
        String decrypted = new String(MyUtils.decryptAndDecode(portFile.text))
        if (decrypted.isInteger()) {
          String ping = "ping"
          client = new Socket((String) null, decrypted.toInteger())
          client.setSoTimeout(1000)
          client << ping //send ping to the server
          client.inputStream.withStream {InputStream sis ->
            sis.eachByte(256) {byte[] buff, int read ->
              ret = (ping != new String(buff, 0, read)) //response same? we are not launchable!
            }
          }
        }
      } catch (Exception e) {
        log.debug("Couldn't determine server running. ${e}")
      } finally {
        if (client) client.close()
      }
    }

    if (ret) { //we're launchable, now grab hold of a server connection and encrypt port to file
      Thread.startDaemon("isLaunchableServerSocket") {
        ServerSocket serverSocket = new ServerSocket()
        serverSocket.bind(null) //creates a dynamic port
        portFile.text = MyUtils.encryptAndEncode(serverSocket.localPort.toString().getBytes())
        while (!serverSocket.isClosed()) {
          Socket socketConn = null
          try {
            socketConn = serverSocket.accept() 
            socketConn.inputStream.withStream {InputStream sis ->
              sis.eachByte(256) {byte[] buff, int read ->
                String msg = new String(buff, 0, read)
                if ("close" != msg) {
                  socketConn.outputStream << msg //just ping it right back
                } else {
                  serverSocket.close()
                }
              }
            }
          } catch (java.net.SocketException e) {
            log.debug("Shutting down local socket.")
            portFile.text = "" //wipe out the text of the port file
          } finally {
            if (socketConn && !socketConn.isClosed()) socketConn.close()
          }
        }
      }
    }
    return ret
  }

Exciting News - Groovy / Grails Project Getting Some Press

As I've mentioned in previous blog posts, the company I work for has been working quite diligently on developing a device and software system that will be used in clinical trials to collect and process ECG data.  The ultimate goal is to help ensure safety of drugs during the development process; we believe that the device and process we've created will provide much higher quality data in a much shorter amount of time.  At its core, the device records 5 minutes of ECG data, and 30 seconds of voice data.  The voice data is used to perform biometric matches to ensure demographic data integrity, and to prevent fraud.  Data transmission is done via HTTPS, and is processed asynchronously on the server.  Once processed on the server, the data is available in real-time for review by selected cardiologists.  Upon performing the review, the analyzed data is available via web interface for real time review by study sponsors and regulatory authorities.

Speaking of regulatory authorities, we've recently received 510K approval from the FDA that we can move forward with this project!

With all of the marketing / general news, one thing that probably won't be communicated is the geek stuff happening behind the scenes. This project heavily leverages Groovy / Grails throughout the entire system work-flow.  Simply put, the following tools have greatly contributed to the success of this project:

• Groovy Swing Builder for the richer UI portions
• Groovy Scriptom for integration with Windows COM dependencies (low level ECG processing with software developed by our Italian friends)
• Grails for the server side stuff, heavily leveraging the Quartz plugin for asynchronous processing

Recent news postings:

• Spaulding IQ One-Button Handheld ECG Device Receives FDA Approval
• Spaulding thinks small to succeed

A Groovy way to track HttpClient Uploads

Recently, I had to develop a client-server application that leverage Commons HttpClient to perform file uploads.  With the upload process, I wanted to provide the user a nice progress bar that showed how much of the data (up to 2 GB) was processed.  There are two obvious approaches:

1. Spawn a thread on the client that constantly checks the server to see how many bytes have been written on the server using something RESTful in nature
2. Register a call-back mechanism that allows tracking the upload progress

 I chose the latter, and leveraging Groovy I/O stuff makes it a snap:

1) Create an implementation of Observable

/**
 * A utility that contains all of the client functionality needed to perform the upload tasks.  
 * This class is observable on the upload progress.
 *
 * @author Brock Heinz
 */
class ClientUtility extends Observable {

  /**
   * Wrapper around observable
   *
   * @param totalBytesWritten - number of bytes written to a stream to the server
   */
  void fireUploadProgress(int totalBytesWritten) {
    setChanged()
    notifyObservers(totalBytesWritten)
  }

  def doUpload(File f) {
    //we'll come back to this later in post
  }
}

2) Create an implementation of FilePart that overrides the sendData() method

/**
 * A wrapper around the commons httpclient FilePart object.  This class overrides the 
 * streaming portion so that we can track upload progress as we write the file to the 
 * server stream
 *
 * @author Brock Heinz
 */
class ProgressFilePart extends org.apache.commons.httpclient.methods.multipart.FilePart {
  private ClientUtility observable
  private final int uploadBufferSize

  /**
   * @param name the name of the part
   * @param file the file to upload
   * @param observable class that wants to be notified of change during upload
   * @param uploadBufferSize optional parameter that sets the number of bytes read before writing
   */
  ProgressFilePart(String name, File file, ClientUtility observable, int uploadBufferSize = 16384) 
    throws FileNotFoundException {
    super(name, file)
    this.observable = observable
    this.uploadBufferSize = uploadBufferSize
  }

  /**
   * Proxy the method that streams data from the file to the underlying socket stream to server.  
   * This method fires a change on each buffered read to the class' observer.
   *
   * @param os the stream to the server
   */
  protected void sendData(OutputStream os) {
    if (lengthOfData() == 0) return
    int totalread = 0
    getSource().createInputStream().withStream {InputStream is ->
      is.eachByte(uploadBufferSize) {byte[] buffer, int read ->
        os.write(buffer, 0, read)
        totalread += read
        this.observable.fireUploadProgress(totalread)
      }
    }
  }
}

3) Register an Observer, and handle observed callbacks

class UploadDataPanel implements Observer {
  UploadDataPanel(def clientUtility) {
    clientUtility.addObserver(this)
  }
  void update(Observable observable, Object o) {
    //update the UI's progress bar
    swingBuilder.doLater { swingBuilder.uploadProgressBar.setValue(o) }
  }
}

4) Put it all together

  //leverage the progress part
  def doUpload(File f, String postUrl) {
    def httpClient = new HttpClient()
    def post = new PostMethod(postUrl)
    def partArray = [new ProgressFilePart("myfile.zip", f, this)] as Part[]
    post.setRequestEntity(new MultipartRequestEntity(partArray, post.getParams()))
    try {
      httpClient.executeMethod(post)
    } finally {
      post.releaseConnection()
    }
  }

Java WebStart 'offline-allowed'

Wow... been going crazy on this one for a few days.  I have an application that is being delivered via JNLP to run under WebStart.  As a requirement, the application has to run online or offline.  When offline, the data that is to later be transmitted will sit in a local 'queue'.  Once the system comes back online, all of the data in the queue will be uploaded.

I just, for the life of me, could not get this to work!  I checked, double-checked, and tripled-checked the format of my JNLP code.  Nothing seemed to work.  Finally, I decided to look at the JNLP returned from my server (Grails controller generated), and then I saw it!  Cache control headers were being set, telling clients NOT to cache.  It turns out the Java process that downloads the JNLP (javaws) respects that cache setting!  The problem wasn't with the JNLP, but rather the HTTP headers.  Problem solved :)

Here's the trivial script I wrote using Groovy and Commons HttpClient to snoop the headers:

1:  import org.apache.commons.httpclient.HttpClient  
2:  import org.apache.commons.httpclient.methods.GetMethod  
3:  def client = new HttpClient()  
4:  def get = new GetMethod("http://localhost:8080/myapp/jnlp/test")  
5:  try {  
6:   println client.executeMethod(get)  
7:   println get.responseBodyAsStream.text  
8:   get.responseHeaders.each { println it }  
9:  } finally {  
10:   get.releaseConnection()  
11: }  

... one final thought.  If you're generating your JNLP dynamically, make sure you set a Last-Modified response header.  WebStart seems to use that then when determining if the JNLP file has changed, and should thus check for JAR updates.  My suggestion, use the most recently updated JAR that is specified in your JNLP as the value for the Last-Modified header:

def jarFile = findFromExpandedWar(servletContext)
def gmt = TimeZone.getTimeZone("GMT")
def modified =  new Date(jarFile.lastModified())
def pattern = "EEE, dd MMM yyyy HH:mm:ss zzz"
response.setHeader("Last-Modified",  DateFormatUtils.format(modified, pattern,  gmt))

You can find the client JAR files by using ServletContext's getRealPath() method.