Quick Updates - Development Focus

I apologize for the lack of any new posts over the past few months. I'd been suffering from some health issues that limited my ability to focus on technical matters. It has gotten better now, and I am trying to do some catching up.

It has recently occurred to me that my background of working with HPC tools and codes over a wide range of HPC software stacks and platforms, had left me with a development experience that is ad-hoc and poorly structured. This realization was coupled with the understanding that I now only possess the Mac OS X development environment as the only real convenient platform to work with. I can also do some work on my Windows machine via minGW, but it is really set up as my gaming platform. Finally, I do have Ubuntu on Virtualbox on my MacBook Pro on which to do Linux development, but that is fairly inconvenient. My living environment is also limiting any idea that I pick up a dedicated Linux box as an alternative. Finally, I realized that Mac OS X Darwin is not merely a Linux/Unix variant, but has its own set of internal design choices and implementation that make any naive approach to tool development from a Linux-centered perspective unproductive.

As such, I plan on taking a step back after the long hiatus to properly acquaint myself with Mac OS X internals from the perspective of a tool developer. Over the next few months, I will attempt to document my experiences and what I have learned as I try to implement common HPC tools design idioms. These idioms will include basics like attaching/coupling tool runtimes with user code/binaries at varying degrees of transparency. They will also include library wrapping and override idioms. These idioms will perhaps for the basis for my next article, to give readers a sense of where I am coming from with regards to my own software development background involving tools.

Getting Started with Cmake by Kitware

Having worked with HPC codes for the better part of 10 years, I am a little ashamed to say that I am only just starting to learn to use cmake. Better late than never, however.

This post serves as a cmake primer of sorts. I was very much brand-new to cmake, and wanted to start with a very simple example.

Unfortunately, the cmake tutorial by Kitware is actually a part of their book "Mastering Cmake," and not really designed to be read standalone. As such they started with a simple example, but proceeded to add complexity to that example without explaining (I assume it had been covered in the preceding chapters of the book) how one uses cmake to build the binary executables for that example. Anyway, here it is:

http://www.cmake.org/cmake/help/cmake_tutorial.html

It wasn't too difficult to figure out that one just ran "cmake ." where the CMakeLists.txt file was located, followed by "make." However, that had created a mess of temporary files at the folder. The latest forum posts online still indicate that no "cmake clean" feature exists for the reverse process. What I desired was to keep only the "true" sources for archive in a code repository. For that, I needed to apply cmake's support for out-of-source builds. This idiom for building tools/applications separates the source code from the build tree, with the tree's attendant temporary files like object files, temporarily generated code, etc. I believe this is the only idiom serious code developers should use for their build systems. It really is best-practice.

To find out how cmake supports out-of-source builds, I found a good (but a little bit complicated) primer on:

http://web.cs.swarthmore.edu/~adanner/tips/cmake.php

With these instructions, I was able to construct a very simple out-of-source example using Kitware's first tutorial code, combined with the necessary structure. This included two shell scripts to avoid some steps in this idiom which I felt introduced potential sources for error, which can mess up the folder.

My top level folder:

CMakeLists.txt
app
build.sh
clean.sh

The CMakeLists.txt file contains only the preamble, along with how to locate the source:

cmake_minimum_required (VERSION 2.6)
project (OutOfSourceTutorial)

add_subdirectory(app)

The app folder holds the source code file tutorial.C, and the CMakeLists.txt file which tells cmake how to build tutorial.C:

add_executable(Tutorial tutorial.C)

The build.sh script sets up the build folder, and invokes cmake on the top-level CMakeLists.txt file from the build folder. make can then be invoked to build the executable Tutorial. These are the steps which I found to be potential for error if they had to be done manually:

#!/bin/bash                                                                                        
mkdir build
cd build
cmake ..
make

Notes:

1. I had initially named the "app" folder "src," which was intuitive as I had always developed my codes with a "src" folder to a hold pure source code. However, the cmake implementation of this idiom names a build sub-folder the exact same name as its source counterpart. This meant a "src" folder would appear under "build." This offended my senses. I did realize however, that the idiom intended the source folder to be named after the application/tool (or part thereof.) After all, if the folder is already meant to hold only the source files under this idiom, it would be an oxymoron to call it "src."

2. The binary executable Tutorial is built into build/app rather than my expectation of something like build/bin. I am assuming this is because the developers of cmake did not wish to enforce any naming conventions, and that Cpack would take care of creating the necessary include, lib, bin, and man folders for Linux systems that I am familiar with.

Personal HPC Cloud Computing Options

While in the middle of my career transition, it occurred to me that I had been fortunate to have had access to (essentially) unlimited supercomputing facilities. It also occurred to me that I had taken all of these for granted. Whatever I do in the future, I would like the option to do useful work in HPC without having to own a parallel machine of my own or to acquire grants for special access to a scientific resource.

It turns out I might just be able to, and relatively cheaply too!

1) The most promising: Penguin Computing POD services -
http://www.penguincomputing.com/Services/HPCCloud/POD

This seems to have everything I might want in a personal (non-virtualized, even!) HPC platform to conduct any of my own work on my own time. Some details are missing (e.g., how far I can scale) but I suppose I can find out when I inquire for access.

2) Hewlett Packard also offers semi-HPC-like services on their public cloud -
https://www.hpcloud.com/

Unfortunately, the last time I checked, the scaling is limited to 8 cores and the compute environments are virtual. That would be no different from myself acquiring a many-core chip for my own compute platform.

That is it for now, but I am certain I will see more options as I dig deeper.

Linux Signalling Pains

Attempts to make sense of it all for the general collection of samples from Timer Interrupts.

Edit 6/19/2012: Given the many issues that need to be addressed, I've decided it would be clunky to attempt to complete them here. I will cover each individual topic piece-meal in subsequent posts.

This article will be work-in-progress for now. There are many things that I have explored (and conveniently forgotten), many things I have yet to explore and many more things that need to be fitted together to make any coherent sense of the matter. Portability issues play a big role in this.

Useful information sources:
Signal(7) - linux.die.net/man/7/signal
Thread Local Storage - www.akkadia.org/drepper/tls.pdf
Signal Re-entrancy - http://www.delorie.com/gnu/docs/glibc/libc_493.html

At the top level, THREE things will need to be addressed for our purposes:

1. The establishment of timer-interrupts. I currently know of two ways of doing this: through the setitimer call (sys/time.h); or through the timer_create call (signal.h, time.h, link with -lrt).
2. The handling of signals and signal masks. Choices of signals for this purpose are SIGPROF or SIGALRM. Masks need to be handled as necessary through sigprocmask. Edit 6/19/2012: sigprocmask is probably not sufficient for our needs. The alternative is signal(SIGPROF, SIG_IGN). I believe the trouble with the latter is that it applies to the entire process, so very careful handling with respect to threads is required.
3. Working with threads. Considerations have to be made for threads like the use of pthread_sigmask and the not-so-portable use of thread local storage like __thread tls.

 That is it for now. Will attempt to get a more complete picture later.

Exascale Computing: A ballpark guide to an epic journey.

The HPC community is taking concrete steps to conduct the necessary research to take us into the Exascale computing era. This post is really a reminder to myself about the scales and challenges we, as research scientists in the field, will face to varying degrees when Exascale computing comes upon us.

So, what scale of computing are we talking about? As a ballpark figure (ignoring the finer points about FLOPS vs OPS as a measure of computing power), we are talking about 1,000,000,000,000,000,000 (or 10 to the 18th) computation operations per second.

What does this mean in terms of today's technology? Well, one of the common processor chips in use today in HPC is AMD's Opteron 6-core chip with a theoretical peak performance of 10.4 GFlops per core. 100 million of those cores buys us approximately an EFlop. We are now in an era of improving performance by placing more cores on a chip rather than making each core go faster. I believe it is reasonable to believe that theoretical peak performance would not go past 15.0 GFlops per core (current roadmaps appear a little fuzzy about the details, so I could be wrong), so we could go down to just 60-70 million cores to achieve an EFlop. In terms of hardware, the challenge is to connect them efficiently and keep latency low. This may be helped by the expected increase in the number of cores per processor chip (Intel's larabee was planned for 24 cores) in addition to improvements in high-speed interconnects for traditional HPC setups. Power will also be a serious issue. Today, 40W is sometimes considered "energy sipping". However, if you think about it, even if each core "sips" only 1W of power, 100 million of them would eat a whopping 100 MW. At 40W, this would mean 4GW of power. This figure even ignores the cooling required to combat the heat generated by the chips! If this is not addressed, we will need about 4 nuclear reactors to be able to provide the necessary power (some figures I've seen indicate that a modern nuclear reactor can generate up to 500 MW of power).

There is some hope, however, if one takes into account accelerator architectures. These could, for various specialized types of tasks, dramatically increase the performance of each of their cores (some existing literature uses accelerator cores and regular CPU cores interchangeably). Energy consumption per core would also be affected if one uses these definitions. For example, there are many many GPU cores in a single GPU card. I do believe scientists in our field has already recognized these accelerator architectures as being essential to the success of exascale computing.

Using the above assumptions about the hardware, what then needs to be considered for the software? Instead of looking at the general field of software in HPC, I would like to now focus on my sub-field of performance analysis of HPC applications. The very first thing that stares me in the face would be the core-counts. Traditionally, the performance data captured on each core conveniently represents the most accurate description of "what happened?". In the above scenario, we would now have to deal with information coming from 100 million cores. Do we keep 100 million files? Do we merge them? Do we try to summarize the information by various compression techniques (as I have explored in my Phd thesis)? The second challenge would be the massive number of events we would have to deal with. If my hypothesis about core speeds not climbing much faster is true, I would be grateful. For, if they did skyrocket, the same events would happen at a much higher frequency. That would in turn mean, if memory speeds fail to keep up with processor speeds (as they have traditionally failed to), recording these events would result in increased overheads and perturbation. Even if this did not happen, I would still have to deal with the sheer numbers for, in-lieu of speed, the application would be pumping the performance information out in parallel. The I/O subsystem had then better catch up if buffers get filled. Finally, what of the analysis? The new challenges I see, even if we do preserve all the details of performance data, is making sense of the increased heterogeneity of the events and the increased complexity of event interactions and how they affect performance. For example, communication between cores on the same chip should be vastly superior to communication between cores out-of-chip. The increased number of cores per chip could also make performance more sensitive to memory access, since the bandwidth between chip and memory gets tighter (given assumptions). The nature of GPGPU processing also adds new scenarios where performance may be sensitive to the way code is written, data shared or the datatypes used.

This is exciting stuff and I look forward to dealing with these challenges as we march forward.

Dynamic Spanning Trees in HPC

This entry will be from the perspective of the Charm++ runtime and is intended to serve as a basis for wider discussion on the use of spanning trees, possibly dynamic, in other HPC software/middleware systems. As they stand, these are my personal thoughts and speculation. They are more questions than answers. Hopefully, as time goes by, they become more answers than questions.

I assume the reader (myself included) understands the idea behind a spanning tree (and minimum spanning tree). In Charm++, we tend to construct spanning trees over the object communication graph in Charm++ applications. These spanning trees then serve as the backbone for broadcasts and reductions, where the root can be any vertex of the spanning tree.

As I understand it, most other HPC runtimes use binary trees for their broadcast and reductions. If I am not wrong, this is because they do not need to deal with the dynamic nature of their applications. In Charm++, objects can migrate. As such, we have a dynamic runtime environment. Broadcasts and reductions happen in the background of application computation and other communication. I do not, however, know if spanning tree construction in Charm++ takes existing communication graph and volume into account. However, if broadcasts and reductions are to be efficient, they need to be woven around more static communication patterns. Hence, the construction of a minimum spanning tree, using the application's more static object communication volume as edge-weights.

So, the questions, I suppose are:

1. Are parallel spanning trees useful in general HPC software/middleware?

2. Can our experience with spanning trees in Charm++ be directly applicable to other HPC systems that attempt to provide dynamic application behavior?