Craig Ulmer

FAODEL Overview Paper

2018-06-11 faodel pub

Patrick, Scott, and Jay put together an overview paper about what we're doing in FAODEL. It was presented at SIGARCH/SIGHPC's ScienceCloud18 workshop.


Composition of computational science applications, whether into ad hoc pipelines for analysis of simulation data or into well-defined and repeatable workflows, is becoming commonplace. In order to scale well as projected system and data sizes increase, developers will have to address a number of looming challenges. Increased contention for parallel filesystem bandwidth, accomodating in situ and ex situ processing, and the advent of decentralized programming models will all complicate application composition for next-generation systems. In this paper, we introduce a set of data services, Faodel, which provide scalable data management for workflows and composed applications. Faodel allows workflow components to directly and efficiently exchange data in semantically appropriate forms, rather than those dictated by the storage hierarchy or programming model in use. We describe the architecture of Faodel and present preliminary performance results demonstrating its potential for scalability in workflow scenarios.


  • IWAC Paper Craig Ulmer, Shyamali Mukherjee, Gary Templet, Scott Levy, Jay Lofstead, Patrick Widener, Todd Kordenbrock, and Margaret Lawson, "Faodel: Data Management for Next-Generation Application Workflows", ScienceCloud'18, June 2018.

Blimp Tracking Success

2018-06-02 rf planes

A little over a week ago a friend of mine that knows I track planes called me up to tell me he saw the Goodyear Blimp flying over Livermore. When I got home I went to my Pi-aware flight tracker to see if I could spot anything. Nope- nothing was on the current map and my logs didn't have any hits for the ICAO numbers Goodyear has registered with the FAA (their blimps have tail fins N1A to N7A). While I was disappointed, I wasn't too surprised- the one time I did see a blimp on the tracker it wasn't providing position info. I figured the one my friend saw had already landed, and that my logs weren't observing it because I only record planes with positions.

I had the following Friday off so I took some time to poke around a little more. I found someone had posted a video on youtube showing blimp N2A landing in Livermore earlier in the week. That helped me figure out which ICAO id to look for (N2A is A18D51). Amy called to let me know that she'd spotted it while she was driving to Dublin. I checked the tracker again and got a nice surprise- in addition to picking up its transmissions, there were enough pi-aware users in the area to determine its location via MLAT. So far the tracker always reports the blimp's altitude as being "on ground". Flight Aware says its somewhere between 1000 and 2000 feet. I've read that blimps are hard to track because the low altitude makes it difficult to get enough stations with line of sight to do the MLAT. I don't get much range with it- it disappears once it's out around Dublin.

I don't have a good idea of what it's doing out here. Usually the blimp comes out here for sporting events. The Warriors/Cavs championship games started this week, but they're in a closed coliseum. I guess the Giants and A's also have some home games this week. It's definitely been hanging around Livermore a lot though. On the way home from lunch today we stopped at the Livermore airport and watched it land, swap out people, and then take off again. From the tracks I captured it looks like they made a few trips out to Lake Del Valle and back. It was 90 today, so there were probably a lot of people out there cooling it off. It's funny having a giant blimp hanging out in our little town for so long. It's like a giant puppy wandering all over the place.

Antenna Mount

2018-05-13 rf planes

It's taken more than a half year to get back to it but I finally mounted the antenna on the roof. The main problem was that the mount kit they provided was a little small for the pipes I wanted to use on the roof. I wound up buying new u-bolts and cutting my own bracket plates to make it work. It's a little hacky but so far it's stayed put. I also had to buy some longer (LMR240) cable to get to the PI in the garage.

Moving the antenna outside of the garage seems to have had a positive impact on my reception. As the coverage maps show below, I'm seeing a good distribution of planes in all directions, though I'm missing a notch towards the south. The antenna is on the north side of the house and below the top of the roof, so it's likely the roof is interfering.

I'm pretty happy with the range, though. I see a lot more planes in the 150-200mi range now, and even a handful of planes 200-250mi away (usually international flights coming in over the Pacific). Previously, I used to lose planes around Sacramento. Now they disappear around Reno.

FAODEL 1.1803.1 Release

2018-04-23 code

I'm pleased to announce that we've officially open sourced and released FAODEL (Flexible, Asynchronous, Object Data-Exchange Libraries) on Github. The 1.1803.1 version is a snapshot of all our libraries: Kelpie (a key/blob service), Obbox (an async comm engine), Lunasa (a network memory management unit), NNTI (an RDMA portability library from Nessie), and Webhook (an in-app http server for interacting with your app). It's been copyrighted by NTESS and we've received permission to open source it from the Department of Energy under the MIT license. We even show up in DOE Code now.

This was the first time I've done an open source release at work, so it was an adventure figuring our what we had to do. The initial step was just getting all of our code together in one repo we could export. We wound up merging several repos together and refactoring the build system, which made the whole thing easier to use. We then ran our tests over and over on different platforms until the code was in a stable form that ran everywhere. Once all of that was in order, we started through the legal parts of the release process.

Legal Process

In order to release the software we needed to declare the license we were going to use and do a copyright assertion. My initial instinct with the license was just to do the MIT license since it's simple and open. When I talked it over with the group though, I started to see how the protections provided in other licenses (eg BSD or Apache) might be better for us. The discussions dragged on and got more complicated (at one point someone even roped in a prof at UCSC). I eventually got fed up and decided to go with my initial instinct- I just want people to be able to use the software, so the MIT license is just fine.

The next step in the legal process was figuring out the right way to insert the NTESS copyright message. I see a lot of code these days where they stamp both the copyright and the full license on the top of every source file. It drives me crazy because I hate scrolling through code just to figure out what the API calls are. I read that adding all this junk is not necessary from a legal perspective if you have it all documented in the top directory. However, one of my developers noted that he does appreciate seeing a legal note on the files so he knows where it came from after installation. I agreed that this was useful, and wrote a script to prepend all our source with the 3-line copyright our legal people asked us to put. I also had to mark up directories for third-party libs (eg tcmalloc) that we include but are not the original authors.

Next, I had to do a code review with a reviewer to make sure that nothing bad was going into the release. This linting process meant going through all the code and determining whether there was anything sensitive that could cause problems. In addition to the things I'm used to looking for in these reviews, we had to look for crypto-related things because an open source release has to be treated as an international export. Interestingly, the fact that we reviewed the code got it marked as an export controlled item. For a few weeks there we were technically rated as EAR99, the lowest export control they can place on something. Fortunately, after everything cleared in the process we were changed to a publicly releasable code with no export issues.

DOE Approval

After all the signoffs, the lab submitted the release request to the Department of Energy for approval. The DOE has very positive policies for open sourcing software, so it wasn't much of a surprise that they OK'd NTESS's copyright assertion and open source release of this software. One of the perks of having DOE be involved in the process is that they route your info into gov code databases like DOE Code. According to one of the talks at the ECP meeting this year, we're supposed to be assigned a universal DOI record at some point. We're in the system now, but it doesn't look like the DOI has happened yet.

In any case it's great to be done with the release. I'm not expecting other people to use it, but at least we've got a placeholder now.


The code is now hosted at github:

github:faodel/faodel v1.1803.1

I/O and Data Management for ECP

2018-02-05 net systems pub

As the PI for the data portion of Sandia's ATDM Data and Viz project, I needed to give a status update about our work at the annual all-hands Exascale Computing Project meeting in Knoxville. I put together the below poster, which talks about (1) improvement's we've made to SNL's IOSS mesh database library and (2) our work with FAODEL.


ECP Poster Poster presented at ECP Meeting