tag:blogger.com,1999:blog-4478278507171029082024-02-08T07:14:14.834-08:00Chee Wai's Technical NotesDocumenting the process of making things work, and making things better.Chee Wai Leehttp://www.blogger.com/profile/12854913855936196475noreply@blogger.comBlogger25125tag:blogger.com,1999:blog-447827850717102908.post-76797697949325977362017-05-10T03:19:00.001-07:002017-05-10T03:25:11.412-07:00Work-around for exec() differences between Python 2 and Python 3<div dir="ltr" style="text-align: left;" trbidi="on">
Recently I've had to navigate a Python 2 vs 3 compatibility issue with regard to the changes in semantics (?) for the exec() call. I'll note here that I do not yet fully comprehend all of the deep issues and subtleties involved here, so if anyone could enlighten me please do so in the comments. I am also not certain that what I am implementing in order to achieve my goals is in fact a well-established (or even reasonable) coding idiom.<br />
<br />
Anyway in a project I am working on with the OpenWorm Foundation, I encountered a scenario where it was beneficial to parameterize a oft-repeated section of code. In a GET request from Django, I'd loop through all of our supported field types, and invoke a function to process each field type if that field's signature shows up in the GET request.<br />
<br />
The call to filter on a field in Django requires that the field type be a part of the code text - something like - <b><i>filter(<myfield>__icontains=searchString)</myfield></i></b>. The <i><b>searchString</b></i> term is not a problem since it refers to a variable name. <i><b>myField</b></i> however is part of the code text, and cannot be a variable string. So to achieve the parameterization I desired, I had to encapsulate the code in an exec() call like so - <i><b>exec('filter(' + myField + '__icontains=searchString'))</b></i>;<br />
<br />
A quick caveat - the above pseudo-code fragment will work just fine in both Python 2 and 3. It was used to build the context for my motivations for writing code of this nature. The real problem arises when I attempt to assign code of that nature to a local variable in a loop within the function, something akin to an accumulation operation. In Python 2's case, the code will work just fine as intended - the exec() call is treated as a in-place statement. In the case of Python 3, exec() is a function but there are some rules governing the way scoping works that I do not yet fully understand. Because of those rules, direct assignment to local function variables will not work.<br />
<br />
To illustrate here is a code fragment I wrote which more or less captures the nature of what I was trying to achieve in the production code:<br />
<br />
<div dir="ltr" style="text-align: left;" trbidi="on">
<script src="https://gist.github.com/cheelee/3467692691c8bdb19e9b5013237429e8.js"></script>
</div>
In this case, the output looks like this:<br />
<br />
<div dir="ltr" style="text-align: left;" trbidi="on">
<script src="https://gist.github.com/cheelee/4615a7fd3e9c3fd42fa4977adb107c9c.js"></script>
</div>
<br />
There is a weird bug where the number doesn't come out right in the "correct" Python 2 case when "*" was supplied to the code, but I don't think that should distract us from the main problem here. Python 2 will report the expected result, but Python 3 won't.<br />
<br />
The workaround appears to be to assign into a construct like a list. Somehow exec() will allow mutable variables to be modified while properly scoped and referenced like in the following code:<br />
<br />
<div dir="ltr" style="text-align: left;" trbidi="on">
<script src="https://gist.github.com/cheelee/d39a0eb551a0e7b2a8bf6f7d0b1b6a77.js"></script>
</div>
Now Python 2 and Python 3 agrees on the output and behavior:<br />
<br />
<div dir="ltr" style="text-align: left;" trbidi="on">
<script src="https://gist.github.com/cheelee/22db3e9edaddae15486ad51f8c24fd8e.js"></script>
</div>
</div>
Chee Wai Leehttp://www.blogger.com/profile/12854913855936196475noreply@blogger.com0tag:blogger.com,1999:blog-447827850717102908.post-31519203144945759182016-08-19T09:16:00.003-07:002016-09-01T07:54:09.870-07:00Public Digital Credentials via Badge List<div dir="ltr" style="text-align: left;" trbidi="on">
In the course of my recent volunteer work with the OpenWorm community, I've had the opportunity to explore the use of Badges as a community tool to establish digital credentials for members of the community.<br />
<br />
The basic concept of Badges comes from an education perspective. The idea is to permit an open and transparent method of acknowledging one's online course credentials after completing courses, tutorials, and the like. For organizations like OpenWorm, it has the secondary benefit of being a way to identify expert mentors whom new volunteers could approach for help in appropriate technical domains.<br />
<br />
As an open science organization, OpenWorm (<a href="http://openworm.org/">http://openworm.org</a>) has started to make use of Badge List (<a href="https://www.badgelist.com/OpenWorm">https://www.badgelist.com/OpenWorm</a>) for this purpose. And the purpose of this blog post was simply to test the embedding of earned badges from Badge List into a blog. I figured the prior long exposition on badges and digital credentials might make this a more meaningful post :).<br />
<br />
<iframe frameborder="0" height="525" src="https://www.badgelist.com/openworm/hodgkin-huxley-tutorial-graduate/u/cheewai1972.embed" width="100%"></iframe><br />
<br /></div>
Chee Wai Leehttp://www.blogger.com/profile/12854913855936196475noreply@blogger.com0tag:blogger.com,1999:blog-447827850717102908.post-36909809682531793132016-06-15T06:15:00.003-07:002017-05-10T03:27:15.747-07:00Testing Code Embedding via Gists<div dir="ltr" style="text-align: left;" trbidi="on">
<script src="https://gist.github.com/cheelee/acb4c915bd1251472cfbe8c9ae42b2fa.js"></script>
My source for this information:<br>
<a href="http://boddhisattvadedicates.blogspot.com/2013/09/testing-code-sample-gist.html" target="_blank">http://boddhisattvadedicates.blogspot.com/2013/09/testing-code-sample-gist.html</a><p>
The Stackoverflow entry with actual HTML embedded code:<br>
<a href="http://stackoverflow.com/questions/18788724/issues-adding-github-gist-to-my-blogusing-google-blogger/21355714#21355714" target="_blank">http://stackoverflow.com/questions/18788724/issues-adding-github-gist-to-my-blogusing-google-blogger/21355714#21355714</a><p>
<br /></div>
Chee Wai Leehttp://www.blogger.com/profile/12854913855936196475noreply@blogger.com0tag:blogger.com,1999:blog-447827850717102908.post-21312353910447953652015-07-12T20:36:00.002-07:002015-07-14T08:05:57.909-07:00Automatic Performance Profiling for Python with cProfile<div dir="ltr" style="text-align: left;" trbidi="on">
<script src="https://cdn.rawgit.com/google/code-prettify/master/loader/run_prettify.js"></script>
This is extensively documented online in a lot of places, but I figured it is worth spreading some more. Plus I think I may have some additional perspectives to add from my own point of view.<br />
<br />
<div style="text-align: left;">
<u><b>
Some Motivating Background</b></u></div>
This post is motivated by some work I had been doing for the <a href="http://www.openworm.org/" target="_blank">OpenWorm</a> project, and one of the things that had been desired was a quick way to generate and manage performance data for the purposes of performance regression on our test codes. Of course this is not restricted to Python codes, but as it turns out some of our earlier test codes are in Python, and Python's cProfile is one of the low-hanging fruits available for the automatic gathering of basic (and maybe more sophisticated - I'll elaborate later) performance profiles for Python.<br />
<br />
<div style="text-align: left;">
<u><b>
Available Online Documentation</b></u></div>
Profiling and cProfile documentation can be found from the Python documentation here:<br />
<br />
<a href="https://docs.python.org/2/library/profile.html">https://docs.python.org/2/library/profile.html</a><br />
<br />
Googling for "cProfile" should also result in a decent number of technical blog posts discussing this tool.<br />
<br />
<div style="text-align: left;">
<u><b>Usage</b></u></div>
<div style="text-align: left;">
Usage is pretty straight forward and in its most basic form requires no changes to our python code base. One simply specifies the inclusion of the cProfile module in the command line with an output file parameter. These changes are highlighted in bold in the example below:</div>
<div style="text-align: left;">
<i><br /></i></div>
<div style="text-align: left;">
<i>> python <b>-m cProfile -o myPerfProfile.out</b> myPythonCode.py</i><br />
<br />
The output file <i><b>myPerfProfile.out</b></i> is a binary file. Apparently the format is not 100% documented. As I'll elaborate later, it may contain more information than simply a flat function performance profile of your python code. It is also an option to specify an output file to the cProfile module. The result of not exercising the option is a simple text output to stdout of a flat performance profile. While this is kind of convenient, it isn't probably what you'd want if you need to perform (even basic) useful performance analysis or regression analysis of your code (the latter is what our group needs.)<br />
<br />
<div style="text-align: left;">
<u><b>Getting What We Want</b></u></div>
If we printed the default text performance profile to stdout, it is displayed as a list of entries sorted by the names of functions invoked. There are several reasons why this is not satisfactory for our purposes:<br />
<br />
<ol style="text-align: left;">
<li>We may only want to see performance information associated with user code (i.e., functions that our Python code invokes directly.) The default output includes performance information associated with all library calls invoked on behalf of user code. <i><b>In this scenario, we probably want the data sorted by cumulative execution time</b></i> (i.e., the sum of time taken exclusively by our functions, and taken by all function calls made by our functions) <i><b>and filtered appropriately</b></i> in order to figure out where most time is spent where our own code is concerned. From a performance analysis perspective, this grants us some basic information from which code-optimization decisions may be reached.</li>
<li>We may want to see performance information in the form of absolute time spent - exactly which leaf functions take the most time. <i><b>In this scenario, we want the data sorted by total execution time</b></i> (also known as exclusive time) <i><b>with no filtering required</b></i>.</li>
</ol>
For our purposes, we would like to start the consideration of our performance regression needs from the perspective spelled out in point 1. The following python script can be used to extract that information from the binary file generated in the prior section:<br />
<br /></div>
<pre class="prettyprint linenums">import sys
import cProfile
import pstats
if len(sys.argv) != 2:
print 'Performance data file required. You have ' + str(len(sys.argv) - 1) + ' arguments\n'
sys.exit()
else:
stats = pstats.Stats(sys.argv[1])
# This configuration generates somewhat more appropriate user-function focused data
# sorted by cumulative time which includes the time of calls further down the callstack.
stats.sort_stats('cumulative','time','calls')
# stats.sort_stats('time','calls')
# This configuration filters the output so only user-functions will show up.
stats.print_stats('PyOpenWorm')
# stats.print_stats()
</pre>
<br /></div>
Chee Wai Leehttp://www.blogger.com/profile/12854913855936196475noreply@blogger.com0tag:blogger.com,1999:blog-447827850717102908.post-19898988676256363532015-05-31T03:49:00.002-07:002015-05-31T03:50:10.248-07:00Some Brief Notes On -finstrument-functions<div dir="ltr" style="text-align: left;" trbidi="on">
Been trying to get back to profiling some of my Mac OS X stuff, and I struggled a little bit to get this working. The main snafu on my part was testing a "simple" code naively. Not really thinking, I did:<br />
<br />
<i><b>gcc -finstrument-functions test.c instrument.c -o test</b></i><br />
<br />
This results in a segfault, which on closer examination via lldb (gdb equivalent on the Mac) would show a very very deep stack trace. The reason this happens is because the instrumentation functions <i>__cyg_profile_func_enter</i> and <i>__cyg_profile_func_exit</i> both also get automatically included in the instrumentation setup.<br />
<br />
There appears to be a non-standard function attribute that can be associated with the 2 instrumentation functions, but others appear to have run into scenarios where compilers would not recognize the attribute, resulting in the same segfault.<br />
<br />
The solution is simple:<br />
<br />
<i><b>gcc -c instrument.c</b></i><br />
<i><b>gcc -finstrument-functions test.c instrument.o -o test</b></i><br />
<br />
but this just tells me the support for profiling in this setup is pretty much an afterthought. Anyway, this really is a note-to-self in case I forget in the future, and need to figure this thing out again.</div>
Chee Wai Leehttp://www.blogger.com/profile/12854913855936196475noreply@blogger.com0tag:blogger.com,1999:blog-447827850717102908.post-31943902654631822392015-01-26T00:32:00.000-08:002015-01-26T01:10:07.898-08:00Mac OS X Latex Support on Yosemite<div dir="ltr" style="text-align: left;" trbidi="on">
I had not intended to jump the gun on documenting Latex over the various other overall stages of my attempt to set up a coherent development environment on a fresh Yosemite machine ... but my preference is to get the documentation down, and properly linked against my master software list first. So here goes.<br />
<br />
<div dir="ltr" id="docs-internal-guid-1fef6b95-255d-638d-2239-768840e8b66c" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">I recalled a great deal of pain when I used Homebrew to install tex-live on my MacBook Pro. Sadly, I do not recall what it was … it was one of those pain-once forget-later scenarios I am determined to eliminate when setting up my development environment on my Yosemite machine.</span></div>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">This online article sums it up (amongst others which basically said the same thing about latex and homebrew): </span><a href="http://tex.stackexchange.com/questions/97183/what-are-the-practical-differences-between-installing-latex-from-mactex-or-macpo" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline;">http://tex.stackexchange.com/questions/97183/what-are-the-practical-differences-between-installing-latex-from-mactex-or-macpo</span></a></div>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">The key takeaway was this article linked in the above:</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<a href="https://github.com/Homebrew/homebrew/issues/1087" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline;">https://github.com/Homebrew/homebrew/issues/1087</span></a></div>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">and the comment from the homebrew people themselves:</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<blockquote class="tr_bq">
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: blue; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">“Installing TeX from source is weird and gross, requires a lot of patches,</span><span style="background-color: transparent; color: blue; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><br class="kix-line-break" /></span><span style="background-color: transparent; color: blue; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">and only builds 32-bit (and thus can't use Homebrew deps on Snow Leopard.)</span><span style="background-color: transparent; color: blue; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><br class="kix-line-break" /></span><span style="background-color: transparent; color: blue; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><br class="kix-line-break" /></span><span style="background-color: transparent; color: blue; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">We recommend using a MacTeX distribution: </span><a href="http://www.tug.org/mactex/" style="text-decoration: none;"><span style="background-color: transparent; color: blue; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline;">http://www.tug.org/mactex/</span></a><span style="background-color: transparent; color: blue; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">”</span></div>
</blockquote>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">However, there seems to be hope in the form of MacTex being available via a Homebrew cask - </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline;">brew cask install mactex</span></div>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">This process has a caveat however:</span></div>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: blue; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">Arya:~ cheelee$ brew cask install mactex</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: blue; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">==> Caveats</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: blue; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">To use mactex, zsh users may need to add the following line to their</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: blue; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">~/.zprofile. (Among other effects, /usr/texbin will be added to the</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: blue; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">PATH environment variable):</span></div>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: blue; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"> eval `/usr/libexec/path_helper -s`</span></div>
<br />
Finally, mactex installs its binaries (pdflatex is now a default symlink to pdftex as a convenience) in /usr/texbin. To re-create the command-line interface I am familiar with, with access to such tools as pdflatex, bibtex, etc ... I needed to create an environment module for mactex/2014 to be loaded with my bash shell as a default (see <a href="http://cwleehpc.blogspot.com/2013/09/the-modules-environment-on-mac-os-x.html">http://cwleehpc.blogspot.com/2013/09/the-modules-environment-on-mac-os-x.html</a>).</div>
Chee Wai Leehttp://www.blogger.com/profile/12854913855936196475noreply@blogger.com0tag:blogger.com,1999:blog-447827850717102908.post-45001234300307676832015-01-16T04:33:00.003-08:002015-01-16T04:39:49.762-08:00Update: New Mac Mini - Baseline Software Stack Setup<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
<div style="text-align: left;">
<span style="font-size: small;"><span style="font-family: inherit;">I've successfully set up a baseline software environment upon which I can proceed to build/install additional tools, and eventually my own software development projects on my Yosemite system.</span></span></div>
<div style="text-align: left;">
<span style="font-size: small;"><span style="font-family: inherit;"><br /></span></span></div>
<div style="text-align: left;">
<span style="font-size: small;"><span style="font-family: inherit;">1. <b>XCode</b> - this is downloaded/installed for free using the App Store application that is shipped with Yosemite. Straightforward process - one point to note is we no longer seem to be required to activate CommandLine Tools as a separate optional feature of XCode.</span></span></div>
<div style="text-align: left;">
<span style="font-size: small;"><span style="font-family: inherit;"><br /></span></span></div>
<div style="text-align: left;">
<span style="font-size: small;"><span style="font-family: inherit;">2. <b>Homebrew</b> - The reason I think this should come second is because I <i>speculate</i> Homebrew's requirement for ruby necessitates getting XCode first. Again, the process is fairly straightforward (information about Homebrew can be found here - <a href="http://computers.tutsplus.com/tutorials/homebrew-demystified-os-xs-ultimate-package-manager--mac-44884" target="_blank">http://computers.tutsplus.com/tutorials/homebrew-demystified-os-xs-ultimate-package-manager--mac-44884</a>) :</span></span></div>
<ul style="text-align: left;">
<li><span style="font-size: small;"><span style="font-family: inherit;">Download via <a href="http://brew.sh/">http://brew.sh</a></span></span></li>
<li><span style="font-size: small;"><span style="font-family: inherit;"><span id="docs-internal-guid-03987d5f-f2ac-fab7-3e72-6921b7aa42ac" style="background-color: transparent; color: black; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"</span></span></span></li>
<li><span style="font-size: small;"><span style="font-family: inherit;"><span id="docs-internal-guid-03987d5f-f2ac-fab7-3e72-6921b7aa42ac" style="background-color: transparent; color: black; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">brew doctor</span></span></span></li>
</ul>
</div>
<div style="text-align: left;">
<span style="font-size: small;"><span style="font-family: inherit;"><span id="docs-internal-guid-03987d5f-f2ac-fab7-3e72-6921b7aa42ac" style="background-color: transparent; color: black; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">3.</span><span id="docs-internal-guid-03987d5f-f2ac-fab7-3e72-6921b7aa42ac" style="background-color: transparent; color: black; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"> <b>Modules Environment</b></span><span id="docs-internal-guid-03987d5f-f2ac-fab7-3e72-6921b7aa42ac" style="background-color: transparent; color: black; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"> - while technically not a part of a baseline software stack, I am including this here as its installation subsequently allows all other software to be conscientiously built either as the default software layer, or as part of a hot-swappable framework (which is what I am after.)</span></span></span></div>
<div style="text-align: left;">
<span style="font-size: small;"><span style="font-family: inherit;"><span id="docs-internal-guid-03987d5f-f2ac-fab7-3e72-6921b7aa42ac" style="background-color: transparent; color: black; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><br /></span></span></span></div>
<div style="text-align: left;">
<span style="font-size: small;"><span style="font-family: inherit;"><span id="docs-internal-guid-03987d5f-f2ac-fab7-3e72-6921b7aa42ac" style="background-color: transparent; color: black; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">Modules has as its prerequisite the Tcl/Tk libraries and headers. I've documented the details of a successful build/install here - <a href="http://cwleehpc.blogspot.sg/2013/09/the-modules-environment-on-mac-os-x.html">http://cwleehpc.blogspot.sg/2013/09/the-modules-environment-on-mac-os-x.html</a></span></span></span></div>
<div style="text-align: left;">
<span style="font-size: small;"><span style="font-family: inherit;"><span id="docs-internal-guid-03987d5f-f2ac-fab7-3e72-6921b7aa42ac" style="background-color: transparent; color: black; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">At this point, I'll just highlight how I intend to (roughly) structure my development environment to use this baseline stack framework.</span></span></span></div>
<div style="text-align: left;">
<span style="font-size: small;"><span style="font-family: inherit;"><span id="docs-internal-guid-03987d5f-f2ac-fab7-3e72-6921b7aa42ac" style="background-color: transparent; color: black; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">Folder structure:</span></span></span></div>
<div style="text-align: left;">
<span style="font-size: small;"><span style="font-family: inherit;"><i><span id="docs-internal-guid-03987d5f-f2ac-fab7-3e72-6921b7aa42ac" style="background-color: transparent; color: black; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">software</span></i><span id="docs-internal-guid-03987d5f-f2ac-fab7-3e72-6921b7aa42ac" style="background-color: transparent; color: black; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"> </span></span></span></div>
<div style="text-align: left;">
<span style="font-size: small;"><span style="font-family: inherit;"><span id="docs-internal-guid-03987d5f-f2ac-fab7-3e72-6921b7aa42ac" style="background-color: transparent; color: black; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"> - <i>software/non-repository </i>(where source packages are kept for building purposes)</span></span></span></div>
<div style="text-align: left;">
<span style="font-size: small;"><span style="font-family: inherit;"><span id="docs-internal-guid-03987d5f-f2ac-fab7-3e72-6921b7aa42ac" style="background-color: transparent; color: black; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"> - <i>software/installed/packages</i> (where built packages are installed for hot-swap)</span></span></span></div>
<div style="text-align: left;">
<span style="font-size: small;"><span style="font-family: inherit;"><span id="docs-internal-guid-03987d5f-f2ac-fab7-3e72-6921b7aa42ac" style="background-color: transparent; color: black; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"> - <i>software/repository/[git|svn|cvs|hg]</i> (where source packages are kept)</span></span></span></div>
<div style="text-align: left;">
<span style="font-size: small;"><span style="font-family: inherit;"><span id="docs-internal-guid-03987d5f-f2ac-fab7-3e72-6921b7aa42ac" style="background-color: transparent; color: black; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"> - <i>software/modules</i> (where module files control how each installed package is loaded/unloaded)</span></span></span></div>
<div style="text-align: left;">
<span style="font-size: small;"><span style="font-family: inherit;"><span id="docs-internal-guid-03987d5f-f2ac-fab7-3e72-6921b7aa42ac" style="background-color: transparent; color: black; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><br /></span></span></span></div>
<div style="text-align: left;">
<span style="font-size: small;"><span style="font-family: inherit;"><span id="docs-internal-guid-03987d5f-f2ac-fab7-3e72-6921b7aa42ac" style="background-color: transparent; color: black; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">The result is fairly satisfying - Homebrew places default software in a space it manages, and is responsible for appropriately overriding the original default software shipped with Yosemite. Modules then allows me to further compartmentalize alternative software installations in <i>software/installed/packages</i>, and allows me to hot-swap these to override Homebrew's default as appropriate.</span></span></span></div>
<div style="text-align: left;">
<span style="font-size: small;"><span style="font-family: inherit;"><span id="docs-internal-guid-03987d5f-f2ac-fab7-3e72-6921b7aa42ac" style="background-color: transparent; color: black; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><br /></span></span></span></div>
<div style="text-align: left;">
<span style="font-size: small;"><span style="font-family: inherit;"><span id="docs-internal-guid-03987d5f-f2ac-fab7-3e72-6921b7aa42ac" style="background-color: transparent; color: black; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">From this baseline, I will continue to document how I layer my working environment according to the classes of software I think are important to me, for example documentation/publication tools like latex and gnuplot; the host of language compilers; support tools for my intended web development environment etc ...</span></span></span></div>
<span id="docs-internal-guid-03987d5f-f2ac-fab7-3e72-6921b7aa42ac" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span></div>
Chee Wai Leehttp://www.blogger.com/profile/12854913855936196475noreply@blogger.com0tag:blogger.com,1999:blog-447827850717102908.post-81973522668793047212015-01-14T00:56:00.003-08:002015-01-14T00:56:46.824-08:00Update: Documenting Fresh Mac Mini Setup For Development<div dir="ltr" style="text-align: left;" trbidi="on">
I bit the bullet the other day, and picked up a brand new top-of-the-line Mac Mini (dual-core i7 with 16Gb of RAM and a 2TB drive). It was about time - my previous most-modern machine, a MacBook Pro is more than 6 years old. It is so full of ad-hoc software stack changes from since graduate school, that I was actually afraid to upgrade it to Yosemite for fear that I may mess something up.<br />
<br />
With this new machine, I want to document every step I take bringing it from a fresh off-the-shelf Yosemite system to a fully-functional and broad-based software development platform that I would like to work with from my point of view.<br />
<br />
What do I mean by "broad-based?" Well, I want to support (by rough order of priority):<br />
<ol style="text-align: left;">
<li> Mac OS X Unix-style application development (XCode + Homebrew + Modules)</li>
<li> Linux/POSIX application development (VirtualBox with Ubuntu)</li>
<li>Web applications</li>
<li>Mobile applications (XCode)</li>
<li>Windows Unix-style application development (mingw)</li>
</ol>
The first two will form the basis of my main software development effort in the context of my primary work on HPC tools and software. <br />
<br />
I want to make use of this machine not just to do good development work, but also to explore ways to structure and improve my own professional workflows. It will also be a platform where I explore ways to structure how I place and organize software and data that can compartmentalize job-related projects, and personal projects.<br />
<br />
The very exercise of documenting how I set up this machine will make use of these ideas (see my previous entry on the use of Google Drive.) The machine will also be an improvement over my laptops for the purposes of a sustained system for automating:<br />
<ol style="text-align: left;">
<li><i><b>Unit testing</b></i>;</li>
<li><i><b>Correctness regression tracking</b></i> via some kind of web-based dashboard similar to Kitware's CDash;</li>
<li><i><b>Performance regression tracking </b></i>via a similar mechanism (CDash has no current support for performance regression tracking as far as I know); and</li>
<li>Semi-automated <i><b>bug management</b></i>.</li>
</ol>
The next few posts I will make will be in the context of this documentation effort. It is mostly meant to be a record useful to me, but I also hope that by putting my thought process down it will be helpful to anyone else interested in a similar endeavor. </div>
Chee Wai Leehttp://www.blogger.com/profile/12854913855936196475noreply@blogger.com0tag:blogger.com,1999:blog-447827850717102908.post-9219193323075334362015-01-10T06:43:00.003-08:002015-01-10T06:43:34.348-08:00A Brief Observation About the Modern Online Development Environment<div dir="ltr" style="text-align: left;" trbidi="on">
I had been trying over the past few months to organize my thoughts around building a structured professional workflow template designed to mitigate one of my greatest professional weaknesses - memory.<br />
<br />
To that end, my approach had evolved from taking brief notes alongside daily to-do lists (on paper.) While I still use a physical note pad for organizing and tracking daily tasks (I'm not sure why I am not comfortable using existing online workflow tools yet,) I had found that Google Drive offers me an excellent way to organize and track the status of my professional work in a hierarchical manner that I found hard with physical notes. This approach seems to work well with my current activities of building software stacks from source for experimentation purposes. Coupled with disciplined use of scripts to capture current progress, having an online structured scratch space that follows me around seems to help.<br />
<br />
I imagine this would change as I try to reason about my workflow with respect to serious code development. I think that will have to include some effective way to quickly track bugs via a tool like Mantis (which I have installed,) but also ways to automate most of the work required to track the code base for correctness and performance regressions over time. A tool like Kitware's CDash comes to mind, but I'll have to see if this can be generalized without requiring ad-hoc meta-tool development the way most research groups do it.<br />
<br />
On a side note - I found it interesting how the modern online infrastructure has changed to allow some of these features (like this blog!) to be available to anyone publicly and for free.</div>
Chee Wai Leehttp://www.blogger.com/profile/12854913855936196475noreply@blogger.com0tag:blogger.com,1999:blog-447827850717102908.post-43220609114838801512014-10-31T05:53:00.000-07:002014-10-31T05:53:06.819-07:00Quick Updates - Development Focus<div dir="ltr" style="text-align: left;" trbidi="on">
I apologize for the lack of any new posts over the past few months. I'd been suffering from some health issues that limited my ability to focus on technical matters. It has gotten better now, and I am trying to do some catching up.<br />
<br />
It has recently occurred to me that my background of working with HPC tools and codes over a wide range of HPC software stacks and platforms, had left me with a development experience that is ad-hoc and poorly structured. This realization was coupled with the understanding that I now only possess the Mac OS X development environment as the only real convenient platform to work with. I can also do some work on my Windows machine via minGW, but it is really set up as my gaming platform. Finally, I do have Ubuntu on Virtualbox on my MacBook Pro on which to do Linux development, but that is fairly inconvenient. My living environment is also limiting any idea that I pick up a dedicated Linux box as an alternative. Finally, I realized that Mac OS X Darwin is not merely a Linux/Unix variant, but has its own set of internal design choices and implementation that make any naive approach to tool development from a Linux-centered perspective unproductive.<br />
<br />
As such, I plan on taking a step back after the long hiatus to properly acquaint myself with Mac OS X internals from the perspective of a tool developer. Over the next few months, I will attempt to document my experiences and what I have learned as I try to implement common HPC tools design idioms. These idioms will include basics like attaching/coupling tool runtimes with user code/binaries at varying degrees of transparency. They will also include library wrapping and override idioms. These idioms will perhaps for the basis for my next article, to give readers a sense of where I am coming from with regards to my own software development background involving tools.</div>
Chee Wai Leehttp://www.blogger.com/profile/12854913855936196475noreply@blogger.com0tag:blogger.com,1999:blog-447827850717102908.post-88452623469569051842014-08-24T02:13:00.001-07:002014-08-24T02:13:56.746-07:00An Overview of An Instance of the Software Production Process<div dir="ltr" style="text-align: left;" trbidi="on">
I had been following a rather well-made youtube channel that highlights a modern game software development process. The channel aims to show how one goes about creating an online MMO game using available tools (as far as I can tell, they use free tools where possible.)<br />
<br />
The following link is a video which from my perspective encapsulates many of the workflows that go into a modern online production software process. The creators had requested to Youtube to disable embedding.<br />
<br />
<span class="watch-title " dir="ltr" id="eow-title" title="MMO Update - Auto Updater Completed!">Video - "MMO Update - Auto Updater Completed!</span>"<br />
<a href="http://youtu.be/YmPNdHS1GXo">http://youtu.be/YmPNdHS1GXo</a><br />
<br />
Coming from an academic environment, where software development and deployment is significantly more adhoc, this is a real eye-opener for me. It is of course debatable if academic codes (even stable production ones) need similar deployment processes that allows such a smooth transition. Personally I'd argue that it does, having gone through much of the pain myself and having reflected on whether the adhoc nature of the codes significantly impact such workflows. I believe they are orthogonal - most academic software development projects already rely on many pieces of these workflows. These include:<br />
<br />
<ul style="text-align: left;">
<li><i><b>code repositories</b></i> to orchestrate modifications from many grad student development sources - these tend to be common to professional software development environments, like the use of git or web interfaces based on git;</li>
<li><i><b>bug tracking and automated/semi-automated unit testing methods</b></i> - these are also common, but anecdotally I have found academic environments to be fairly lax on this aspect. Tools from Kitware like CMake/CTest/CDash might help change this. Even so, I'm still trying to find out if something like CTest could automatically be linked to a bug-tracker like Mantis.</li>
<li><i><b>code packaging and deployment</b></i> - academic environments definitely lag behind professional environments in this department. It may be for good reason since academic codes tend to need to be flexible more than they need to be consistent to their users. In HPC for instance some older code versions tend to work (or work better) on some machine platforms. These forces prompt users of academic codes to be rather particular about specific platform optimizations. Given the platforms tend to be very transient (with 3-5 years operating life) it would probably be problematic (in terms of relative effort) for academic tools to stay highly tuned to ALL the platforms they support in EACH new version. Still, some tools like CPack from Kitware does try to provide a path toward a cross-platform deployment workflow. At the other extreme, MMO games with multi-user interactions pretty much force developers to ensure a simple, smooth, and consistent deployment workflow for all their clients to experience exactly the same game at all times. In the middle, I've noticed other professional tools like Flash Player and operating systems like Mac OS X, with workflows sitting somewhere in-between.</li>
</ul>
In any event, that is what drives my attempt to collect a set of (preferably free) software development tools on my machine, that I can be comfortable with and satisfies the requirements of each of the major parts of the software development workflows over multiple domains - from academic to professional. I think I have something, but am still trying to wrap my head around all of it (like getting some of these pieces to work well with one another.) I wished I had learned this as a student - but I guess there are reasons why some things are taught in vocational schools, and others in a university degree course. I belonged to the latter world, and am only starting to see software from the other side of this "divide." The video I linked had helped me a lot, at least with the big picture and so I would like to share it here.</div>
Chee Wai Leehttp://www.blogger.com/profile/12854913855936196475noreply@blogger.com0tag:blogger.com,1999:blog-447827850717102908.post-62908675901893028712014-08-16T07:04:00.000-07:002014-08-16T07:04:56.024-07:00install_name_tool on Mac OS X<div dir="ltr" style="text-align: left;" trbidi="on">
I'll repost this link to my Leap Motion development blog, but knowing the utility <i>install_name_tool</i> existed many months ago would have saved me much grief trying to figure out why I could never link a standard C++ app to their supplied dylib SDK library except through Xcode.<br />
<br />
I knew they had a library path "@loader_path/libLeap.dylib" which confused the linker, but I had no clue then how to change it (surprisingly, googling the error failed to give any mention to <i>install_name_tool</i>.) Understanding very little of Mac OS X-specific development issues, I made the assumption that dylib libraries worked their own special ways (as <a href="http://en.wikipedia.org/wiki/Mach-O" target="_blank">MACH-O binaries</a>, they actually somewhat do), and that Xcode would know how to deal with that weird path.<br />
<br />
Well, now I know. And kudos to the Leap Devs for releasing examples in plain C++ (and a Makefile) which helped me lots!</div>
Chee Wai Leehttp://www.blogger.com/profile/12854913855936196475noreply@blogger.com0tag:blogger.com,1999:blog-447827850717102908.post-85266236138548317582014-08-11T07:22:00.003-07:002014-08-11T07:24:52.468-07:00Getting Started with Cmake by Kitware<div dir="ltr" style="text-align: left;" trbidi="on">
Having worked with HPC codes for the better part of 10 years, I am a little ashamed to say that I am only just starting to learn to use cmake. Better late than never, however.<br />
<br />
This post serves as a cmake primer of sorts. I was very much brand-new to cmake, and wanted to start with a very simple example.<br />
<br />
Unfortunately, the cmake tutorial by Kitware is actually a part of their book "Mastering Cmake," and not really designed to be read standalone. As such they started with a simple example, but proceeded to add complexity to that example without explaining (I assume it had been covered in the preceding chapters of the book) how one uses cmake to build the binary executables for that example. Anyway, here it is:<br />
<br />
<a href="http://www.cmake.org/cmake/help/cmake_tutorial.html">http://www.cmake.org/cmake/help/cmake_tutorial.html</a><br />
<br />
It wasn't too difficult to figure out that one just ran "cmake ." where the CMakeLists.txt file was located, followed by "make." However, that had created a mess of temporary files at the folder. The latest forum posts online still indicate that no "cmake clean" feature exists for the reverse process. What I desired was to keep only the "true" sources for archive in a code repository. For that, I needed to apply cmake's support for out-of-source builds. This idiom for building tools/applications separates the source code from the build tree, with the tree's attendant temporary files like object files, temporarily generated code, etc. I believe this is the only idiom serious code developers should use for their build systems. It really is best-practice.<br />
<br />
To find out how cmake supports out-of-source builds, I found a good (but a little bit complicated) primer on:<br />
<br />
<a href="http://web.cs.swarthmore.edu/~adanner/tips/cmake.php">http://web.cs.swarthmore.edu/~adanner/tips/cmake.php</a><br />
<br />
With these instructions, I was able to construct a very simple out-of-source example using Kitware's first tutorial code, combined with the necessary structure. This included two shell scripts to avoid some steps in this idiom which I felt introduced potential sources for error, which can mess up the folder.<br />
<br />
My top level folder:<br />
<br />
<i><b>CMakeLists.txt</b></i><br />
<i><b>app</b></i><br />
<i><b>build.sh</b></i><br />
<i><b>clean.sh</b></i><br />
<br />
The <i><b>CMakeLists.txt</b></i> file contains only the preamble, along with how to locate the source:<br />
<br />
<span style="font-size: xx-small;"><span style="font-family: "Courier New",Courier,monospace;">cmake_minimum_required (VERSION 2.6)<br />project (OutOfSourceTutorial)<br /><br />add_subdirectory(app)</span></span><br />
<br />
The <i><b>app</b></i> folder holds the source code file <i><b>tutorial.C</b></i>, and the <i><b>CMakeLists.txt</b></i> file which tells cmake how to build <i><b>tutorial.C</b></i>:<br />
<br />
<span style="font-size: xx-small;"><span style="font-family: "Courier New",Courier,monospace;">add_executable(Tutorial tutorial.C)</span></span><br />
<br />
The <i><b>build.sh</b></i> script sets up the build folder, and invokes <i>cmake</i> on the top-level <i><b>CMakeLists.txt</b></i> file from the build folder. <i>make</i> can then be invoked to build the executable <i><b>Tutorial</b></i>. These are the steps which I found to be potential for error if they had to be done manually:<br />
<br />
<span style="font-size: xx-small;"><span style="font-family: "Courier New",Courier,monospace;">#!/bin/bash <br />mkdir build<br />cd build<br />cmake ..<br />make</span></span><br />
<br />
Notes:<br />
<br />
1. I had initially named the "app" folder "src," which was intuitive as I had always developed my codes with a "src" folder to a hold pure source code. However, the cmake implementation of this idiom names a build sub-folder the exact same name as its source counterpart. This meant a "src" folder would appear under "build." This offended my senses. I did realize however, that the idiom intended the source folder to be named after the application/tool (or part thereof.) After all, if the folder is already meant to hold only the source files under this idiom, it would be an oxymoron to call it "src."<br />
<br />
2. The binary executable <i><b>Tutorial</b></i> is built into <i>build/app</i> rather than my expectation of something like <i>build/bin</i>. I am assuming this is because the developers of cmake did not wish to enforce any naming conventions, and that Cpack would take care of creating the necessary <i>include</i>, <i>lib</i>, <i>bin</i>, and <i>man</i> folders for Linux systems that I am familiar with.</div>
Chee Wai Leehttp://www.blogger.com/profile/12854913855936196475noreply@blogger.com0tag:blogger.com,1999:blog-447827850717102908.post-65387745704845481622014-08-05T02:38:00.000-07:002014-08-05T02:38:07.763-07:00A Quick Update<div dir="ltr" style="text-align: left;" trbidi="on">
It has been awhile. I have noted that my previous posts tended to be disorganized, and I'd like to give this blog some structure. I am starting by creating a tab of "sticky" pages to help organize the material.<br />
<br />
The first page serves as a way for me to keep track of the many software tools that I use, and find essential for my development work on my MacBook Pro. In creating this page, I hope I can also help others decide on the kinds of tools that help make their lives as developers productive.</div>
Chee Wai Leehttp://www.blogger.com/profile/12854913855936196475noreply@blogger.com2tag:blogger.com,1999:blog-447827850717102908.post-72086528166314842972013-09-10T01:26:00.002-07:002015-01-16T04:07:57.566-08:00The Modules Environment on Mac OS X<div dir="ltr" style="text-align: left;" trbidi="on">
Somehow I keep forgetting to archive pertinent information about this, so I forget the details each time I have to do it with a new machine.<br />
<br />
Anyway, here's the link to information courtesy of a group at NASA. The notes seem fairly complete, so I do not see a need to make any modifications as my own set of notes here:<br />
<br />
<a href="https://modelingguru.nasa.gov/docs/DOC-2015">https://modelingguru.nasa.gov/docs/DOC-2015</a><br />
<br />
<b>Updated 1/16/2015</b>:<br />
It would appear additional notes are in order for some relatively new revelations as I tried to get a new Mac Mini started (see <a href="http://cwleehpc.blogspot.sg/2015/01/update-documenting-fresh-mac-mini-setup.html">http://cwleehpc.blogspot.sg/2015/01/update-documenting-fresh-mac-mini-setup.html</a>)<br />
<br />
1. The Tcl library and headers are a prerequisite for Modules. On my older MacBook Pro, this prerequisite was satisfied before I had thought to document any setup process from scratch. The default Tcl/Tk setup on Yosemite does not appear to lend itself easily to satisfying this prerequisite, and so I decided to use Homebrew to do this.<br />
<ul style="text-align: left;">
<li><span id="docs-internal-guid-48932c9b-f298-066a-c39e-e87690b83510" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><i>brew tap homebrew/dupes</i> (Tcl/Tk is not part of the standard repository)</span></li>
<li><i><span id="docs-internal-guid-48932c9b-f298-a580-3d51-5a2bb6d19529" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><i>brew install tcl-tk</i></span></i></li>
</ul>
<span id="docs-internal-guid-48932c9b-f298-a580-3d51-5a2bb6d19529" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">This process issues the following caveat:</span><br />
<br />
<div dir="ltr" id="docs-internal-guid-48932c9b-f29a-12ec-06ff-690fefd2a7b2" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: red; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline;">“This formula is keg-only, which means it was not symlinked into /usr/local.</span></div>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: red; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline;">Mac OS X already provides this software and installing another version in</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: red; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline;">parallel can cause all kinds of trouble.</span></div>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: red; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline;">Tk installs some X11 headers and OS X provides an (older) Tcl/Tk.</span></div>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: red; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline;">Generally there are no consequences of this for you. If you build your</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: red; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline;">own software and it requires this formula, you'll need to add to your</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: red; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline;">build variables:</span></div>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: red; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline;"> </span><span style="background-color: transparent; color: red; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline;">LDFLAGS: -L/usr/local/opt/tcl-tk/lib</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: red; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline;"> </span><span style="background-color: transparent; color: red; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline;">CPPFLAGS: -I/usr/local/opt/tcl-tk/include”</span></div>
<span id="docs-internal-guid-48932c9b-f298-a580-3d51-5a2bb6d19529" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"> </span><i><span id="docs-internal-guid-48932c9b-f298-a580-3d51-5a2bb6d19529" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><i> </i></span></i><br />
<div>
which is then made use of in the following (preliminary) configure line to Modules:</div>
<div>
<br />
<span id="docs-internal-guid-48932c9b-f298-a580-3d51-5a2bb6d19529" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span></div>
<div>
<span id="docs-internal-guid-83c8b8f0-f29b-6bb5-c434-2a36a2c9fa1d" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">./configure --prefix=</span><span id="docs-internal-guid-83c8b8f0-f29e-191b-5a8b-a575d53bca1c" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">$INSTALL_PATH</span><span id="docs-internal-guid-83c8b8f0-f29b-6bb5-c434-2a36a2c9fa1d" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><install_path>/modules-3.2.10 --with-tcl-lib=/usr/local/opt/tcl-tk/lib --with-tcl-inc=/usr/local/opt/tcl-tk/include</install_path></span></div>
<div>
<br />
<span id="docs-internal-guid-48932c9b-f298-a580-3d51-5a2bb6d19529" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span></div>
<div>
This line actually fails, which leads us to point 2.</div>
<div>
<br />
<span id="docs-internal-guid-48932c9b-f298-a580-3d51-5a2bb6d19529" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span></div>
<div>
2. The latest version of Modules does not check for a recent function deprecation in Tcl/Tk. This fact was captured in the following post: <a href="http://sourceforge.net/p/modules/bugs/62/">http://sourceforge.net/p/modules/bugs/62/</a></div>
<div>
<br />
<span id="docs-internal-guid-48932c9b-f298-a580-3d51-5a2bb6d19529" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span></div>
<div>
This forces a final modification to the configure line that works:</div>
<div>
<br />
<span id="docs-internal-guid-48932c9b-f298-a580-3d51-5a2bb6d19529" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span></div>
<div>
<span id="docs-internal-guid-83c8b8f0-f29e-191b-5a8b-a575d53bca1c" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">./configure CPPFLAGS="-DUSE_INTERP_ERRORLINE" --prefix=<install_path>$INSTALL_PATH/modules-3.2.10 --with-tcl-lib=/usr/local/opt/tcl-tk/lib --with-tcl-inc=/usr/local/opt/tcl-tk/include</install_path></span></div>
<div>
<br />
<span id="docs-internal-guid-48932c9b-f298-a580-3d51-5a2bb6d19529" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span></div>
<div>
3. 2 warnings show up during the configure process:</div>
<div>
<br />
<span id="docs-internal-guid-48932c9b-f298-a580-3d51-5a2bb6d19529" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span></div>
<div>
<span id="docs-internal-guid-83c8b8f0-f29e-f3ad-5cee-34dfa176e8a4" style="background-color: transparent; color: red; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">configure: WARNING: will use MODULEPATH=</span><span style="color: red;"><span id="docs-internal-guid-83c8b8f0-f29e-191b-5a8b-a575d53bca1c" style="background-color: transparent; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">$INSTALL_PATH</span></span><span id="docs-internal-guid-83c8b8f0-f29e-f3ad-5cee-34dfa176e8a4" style="background-color: transparent; color: red; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><install_path>/modules-3.2.10/Modules/modulefiles : rerun configure using --with-module-path to override default</install_path></span><span style="background-color: transparent; color: red; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><br class="kix-line-break" /></span><span style="background-color: transparent; color: red; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">configure: WARNING: will use VERSIONPATH=<install_path></install_path></span><span style="background-color: transparent; color: red; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"><install_path><span id="docs-internal-guid-83c8b8f0-f29e-191b-5a8b-a575d53bca1c" style="background-color: transparent; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">$INSTALL_PATH</span>/modules-3.2.10/Modules/versions : rerun configure using --with-version-path to override default</install_path></span></div>
<div>
<br />
<span id="docs-internal-guid-48932c9b-f298-a580-3d51-5a2bb6d19529" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span></div>
<div>
My approach was to ignore them. The rationale behind this was so I allowed each future upgrade of Modules to be installed with their default (and consistent) relative paths. Meanwhile, I could control where my own module files can be hosted using the <i>MODULEPATH</i> environment variable at shell startup (in my case, bash).<br />
<span id="docs-internal-guid-48932c9b-f298-a580-3d51-5a2bb6d19529" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span></div>
</div>
Chee Wai Leehttp://www.blogger.com/profile/12854913855936196475noreply@blogger.com0tag:blogger.com,1999:blog-447827850717102908.post-51141976202968186232013-06-24T14:08:00.001-07:002013-09-10T01:27:13.266-07:00Transition From MacPorts to HomeBrewSo, I've decided to take a dive and get rid of MacPorts and use HomeBrew.<br />
<br />
I will update this systematically as I proceed with the following instructions I found:<br />
<br />
<a href="http://bitboxer.de/2010/06/03/moving-from-macports-to-homebrew/">http://bitboxer.de/2010/06/03/moving-from-macports-to-homebrew/</a><br />
<br />
<b>Update</b>: That seemed to have worked. Now to install Homebrew via:<br />
<br />
<a href="https://github.com/mxcl/homebrew/wiki/installation">https://github.com/mxcl/homebrew/wiki/installation</a><br />
<br />
<b>Edit</b>: That link suggested the more reliable instructions to be found here:<br />
<br />
<a href="http://mxcl.github.io/homebrew/">http://mxcl.github.io/homebrew/</a><br />
<br />
<b>Edit</b>: That seemed to have worked too, but "brew doctor" has some things I need to pay attention to before I can declare this a success.<br />
<br />
It would appear I had forgotten about Fink. Home Brew says I do not need to uninstall it, but my instincts tell me that whatever Fink installed probably needs to be uninstalled or moved into some sandbox.<br />
<br />
The appropriate command appears to be:<br />
<br />
<pre><code>fink purge `fink list | perl -lne '/^\s*i\s+(\S+)/ and print $1'`</code></pre>
<br />
<b>Edit</b>: Okay ... it skipped "Essential and Virtual" packages, and while purging complained of packages in the list that weren't actually installed. What a mess. Let's hope my machine holds after this. Going to remove /sw/bin/fink after this.<br />
<br />
My current solution to fink is to:<br />
<br />
sudo mv /sw ~/MyFinkArchive<br />
<br />
and comment the line in my .profile trying to invoke init.sh in the Fink archive.<br />
<br />
<br />
<br />
<br />Chee Wai Leehttp://www.blogger.com/profile/12854913855936196475noreply@blogger.com0tag:blogger.com,1999:blog-447827850717102908.post-81933684444120131932012-08-12T16:39:00.002-07:002012-08-12T16:39:38.114-07:00Static and Shared object file linkingNo time for any details, but something for me to followup and play around with:<br /><br /><a href="http://stackoverflow.com/questions/1344830/possible-to-build-a-shared-library-with-static-link-used-library">http://stackoverflow.com/questions/1344830/possible-to-build-a-shared-library-with-static-link-used-library</a><br />
<br />
<a href="http://stackoverflow.com/questions/386579/pack-shared-libraries-into-the-elf">http://stackoverflow.com/questions/386579/pack-shared-libraries-into-the-elf</a><br />
<br />
<a href="http://statifier.sourceforge.net/">http://statifier.sourceforge.net/</a><br />
<br />
<a href="http://www.magicermine.com/products.html#feature_comparison">http://www.magicermine.com/products.html#feature_comparison</a>Chee Wai Leehttp://www.blogger.com/profile/12854913855936196475noreply@blogger.com0tag:blogger.com,1999:blog-447827850717102908.post-53782688602489389832012-08-04T19:15:00.001-07:002012-08-04T19:16:46.461-07:00Personal HPC Cloud Computing OptionsWhile in the middle of my career transition, it occurred to me that I had been fortunate to have had access to (essentially) unlimited supercomputing facilities. It also occurred to me that I had taken all of these for granted. Whatever I do in the future, I would like the option to do useful work in HPC without having to own a parallel machine of my own or to acquire grants for special access to a scientific resource.<br />
<br />
It turns out I might just be able to, and relatively cheaply too!<br />
<br />
1) The most promising: Penguin Computing POD services -<br />
<a href="http://www.penguincomputing.com/Services/HPCCloud/POD">http://www.penguincomputing.com/Services/HPCCloud/POD</a><br />
<br />
This seems to have everything I might want in a personal (non-virtualized, even!) HPC platform to conduct any of my own work on my own time. Some details are missing (e.g., how far I can scale) but I suppose I can find out when I inquire for access.<br />
<br />
2) Hewlett Packard also offers semi-HPC-like services on their public cloud -<br />
<a href="https://www.hpcloud.com/">https://www.hpcloud.com/</a><br />
<br />
Unfortunately, the last time I checked, the scaling is limited to 8 cores and the compute environments are virtual. That would be no different from myself acquiring a many-core chip for my own compute platform.<br />
<br />
That is it for now, but I am certain I will see more options as I dig deeper.Chee Wai Leehttp://www.blogger.com/profile/12854913855936196475noreply@blogger.com2tag:blogger.com,1999:blog-447827850717102908.post-69777502556341966452012-07-21T20:46:00.004-07:002012-07-22T12:28:11.894-07:00Ubuntu Setup for Academic publicationIt would appear the default Ubuntu installation has a number of annoyances (nothing major) that need to be overcome before one can start developing and viewing pdf documents with latex.<br />
<ol>
<li>It turns out that Ubuntu's packages for "Canonical Partners" are no longer active by default. I remember the previous version having that available as a default. Instructions for including them (which includes support for Adobe's Acrobat Reader) can be found here:<br /><br /><a href="http://askubuntu.com/questions/89127/how-do-i-install-adobe-acrobat-reader">http://askubuntu.com/questions/89127/how-do-i-install-adobe-acrobat-reader</a><br /><br />In short, these three commands are necessary (if working purely from a terminal):<br /><br />sudo add-apt-repository "deb http://archive.canonical.com/ [distro] partner"<br />
Where [distro] is, for example, lucid, natty, oneiric, precise, quantal<br />
sudo apt-get update<br />
sudo apt-get install acroread<br />
<pre><code> </code></pre>
</li>
<li>It also turns out that "open" is not configured to work out-of-the-box in an initial Ubuntu setup.<br /><br /><a href="http://askubuntu.com/questions/43264/how-to-open-a-pdf-file-from-terminal">http://askubuntu.com/questions/43264/how-to-open-a-pdf-file-from-terminal</a><br /><br />has some information. Of the options, the one I chose (and worked well) was:<br /><br />Adding an "<code>alias open='gnome-open'</code>" to my .bashrc file.</li>
<li>Of course, there's the whole shebang of getting the tetex package, the instructions for which seems to change so frequently that it is really pointless writing it down here. I usually just google for "pdflatex ubuntu" and get it done quickly enough.</li>
</ol>Chee Wai Leehttp://www.blogger.com/profile/12854913855936196475noreply@blogger.com0tag:blogger.com,1999:blog-447827850717102908.post-42396622950932528492012-07-15T19:41:00.003-07:002012-07-22T12:26:42.370-07:00Personal System Backup OptionsA note and brief "discussion" of the various personal backup options I have used and are looking at:<br />
<br />
<b>Mac OS X</b> - hands down <b><i>Time Machine</i></b> [<b style="color: #38761d;">USING</b>]. It is the most convenient game in town, being distributed with the OS. It has overhead that is not really noticeable in time or space, is easy to use for determining what and what not to backup. It is also the only backup system where I have actually had to use. A motherboard failure which killed the graphics display forced me to bring my laptop in for repairs while I was preparing to complete my Phd thesis. I was not going to wait a full week for it to be returned to me, so I promptly bought a new MacBook Pro and tried restoring my old machine state. With some minor problems, <i>it worked like a charm</i>! I was extremely impressed. Not only was my data restored, most of the appropriate parts of the operating system settings came back online smoothly as well.<br />
<br />
<hr />
<br />
<h3>
<u>Linux:</u></h3>
<b>Ubuntu</b> - It looks like Ubuntu 11+ packages a simple backup utility called <b><i>Deju Dup</i></b> [<b><span style="color: #f1c232;">INVESTIGATING</span></b>]. It looks simple and meant for backing up user data (which is fine for Linux nowadays since it is relatively simple to re-install into a fairly usable state from the start). I can see it being valuable for backing up smaller documents and data files, but not necessarily huge experimental data.<br />
<br />
<hr />
<br />
<b>Windows</b> - I have paid for and tried <i><b>Genie 9</b></i> [<b style="color: red;">REJECTED</b>] for a while. It looked very promising at first and came with accolades from various large organizations using it. Unfortunately, the downsides became clear the more I used it on my PC. It suffered from extreme overhead in both space and time. It was set up as a "service" and would kick in the moment the PC was booted up. That might work well for company servers that stay up all the time, but the overhead due to a startup was just overwhelming ... the PC had been observed to stay unresponsive for something like 5 minutes! There was also no convenient way to tell it not to monitor changes to Firefox's cache directory (It is supposed to avoid IE's). As a result, it was constantly trying to back up the changes there and I ended up with only 23% of space left on a 1.82 TB drive after almost a year.<br />
<br />
<b>Windows</b> - I am looking at <a href="http://www.cobiansoft.com/cobianbackup.htm" target="_blank"><i><b>Cobian Backup 11, Gravity</b></i></a>. It is free and looks promising for my purposes.Chee Wai Leehttp://www.blogger.com/profile/12854913855936196475noreply@blogger.com0tag:blogger.com,1999:blog-447827850717102908.post-44955790290773161072012-07-07T23:18:00.002-07:002012-07-22T12:25:52.183-07:00C++ Notes: Was "C+11 and Delegating Constructors"Pretty much a note to self - I did not know this was not allowed until C++11.<br />
<br />
<a href="https://secure.wikimedia.org/wikipedia/en/wiki/C++11#Object_construction_improvement">https://secure.wikimedia.org/wikipedia/en/wiki/C++11#Object_construction_improvement</a><br />
<br />
Available in Java (which I was aware of), C# and D.<br />
<br />
Nice C++ Operator overloading notes:<br />
<br />
<a href="http://courses.cms.caltech.edu/cs11/material/cpp/donnie/cpp-ops.html" target="_blank">http://courses.cms.caltech.edu/cs11/material/cpp/donnie/cpp-ops.html </a><br />
<br />
IOStream manipulators:<br />
<br />
<a href="http://www.arachnoid.com/cpptutor/student3.html" target="_blank">http://www.arachnoid.com/cpptutor/student3.html </a>Chee Wai Leehttp://www.blogger.com/profile/12854913855936196475noreply@blogger.com0tag:blogger.com,1999:blog-447827850717102908.post-16993744592930324342012-06-18T12:12:00.001-07:002012-06-19T20:06:58.739-07:00Linux Signalling Pains<h2>
Attempts to make sense of it all for the general collection of samples from Timer Interrupts.</h2>
<b>Edit 6/19/2012:</b> Given the many issues that need to be addressed, I've decided it would be clunky to attempt to complete them here. I will cover each individual topic piece-meal in subsequent posts. <br />
<br />
This article will be <i><b>work-in-progress</b></i> for now. There are many things that I have explored (and conveniently forgotten), many things I have yet to explore and many more things that need to be fitted together to make any coherent sense of the matter. Portability issues play a big role in this.<br />
<br />
Useful information sources:<br />
Signal(7) - <a href="http://linux.die.net/man/7/signal">linux.die.net/man/7/signal</a><br />
Thread Local Storage - <a href="http://www.akkadia.org/drepper/tls.pdf">www.akkadia.org/drepper/tls.pdf</a><br />
Signal Re-entrancy - <a href="http://www.delorie.com/gnu/docs/glibc/libc_493.html"></a><a href="http://www.delorie.com/gnu/docs/glibc/libc_493.html">http://www.delorie.com/gnu/docs/glibc/libc_493.html</a><br />
<br />
At the top level, THREE things will need to be addressed for our purposes:<br />
<ol>
<li>The establishment of timer-interrupts. I currently know of two ways of doing this: through the <i>setitimer</i> call (sys/time.h); or through the <i>timer_create</i> call (signal.h, time.h, link with -lrt).</li>
<li>The handling of signals and signal masks. Choices of signals for this purpose are SIGPROF or SIGALRM. Masks need to be handled as necessary through <i>sigprocmask</i>. <b>Edit 6/19/2012:</b> sigprocmask is probably not sufficient for our needs. The alternative is <i>signal(SIGPROF, SIG_IGN)</i>. I believe the trouble with the latter is that it applies to the entire process, so very careful handling with respect to threads is required.</li>
<li>Working with threads. Considerations have to be made for threads like the use of <i>pthread_sigmask</i> and the not-so-portable use of thread local storage like <i>__thread tls</i>.</li>
</ol>
That is it for now. Will attempt to get a more complete picture later.Chee Wai Leehttp://www.blogger.com/profile/12854913855936196475noreply@blogger.com0tag:blogger.com,1999:blog-447827850717102908.post-49668518264020123362012-06-17T18:25:00.001-07:002012-07-22T12:27:07.113-07:00A Quick NoteI am changing the nature of this blog to contain notes of a technical or research nature, not necessarily confined to High Performance Computing. It will be based on whatever I am doing (or think I am doing) at the time, so future posts should be reasonably self-organized in chronological order.<br />
<br />
What future posts will likely NOT be are high ideals/ideas with little or no technical merit.Chee Wai Leehttp://www.blogger.com/profile/12854913855936196475noreply@blogger.com0tag:blogger.com,1999:blog-447827850717102908.post-20466628019233785752010-03-15T22:59:00.000-07:002012-07-22T12:27:38.997-07:00Exascale Computing: A ballpark guide to an epic journey.The HPC community is taking concrete steps to conduct the necessary research to take us into the Exascale computing era. This post is really a reminder to myself about the scales and challenges we, as research scientists in the field, will face to varying degrees when Exascale computing comes upon us.<br />
<br />
So, what scale of computing are we talking about? As a ballpark figure (ignoring the finer points about FLOPS vs OPS as a measure of computing power), we are talking about 1,000,000,000,000,000,000 (or 10 to the 18th) computation operations per second.<br />
<br />
What does this mean in terms of today's technology? Well, one of the common processor chips in use today in HPC is AMD's Opteron 6-core chip with a theoretical peak performance of 10.4 GFlops per core. 100 million of those cores buys us approximately an EFlop. We are now in an era of improving performance by placing more cores on a chip rather than making each core go faster. I believe it is reasonable to believe that theoretical peak performance would not go past 15.0 GFlops per core (current roadmaps appear a little fuzzy about the details, so I could be wrong), so we could go down to just 60-70 million cores to achieve an EFlop. In terms of hardware, the challenge is to connect them efficiently and keep latency low. This may be helped by the expected increase in the number of cores per processor chip (Intel's larabee was planned for 24 cores) in addition to improvements in high-speed interconnects for traditional HPC setups. Power will also be a serious issue. Today, 40W is sometimes considered "energy sipping". However, if you think about it, even if each core "sips" only 1W of power, 100 million of them would eat a whopping 100 MW. At 40W, this would mean 4GW of power. This figure even ignores the cooling required to combat the heat generated by the chips! If this is not addressed, we will need about 4 nuclear reactors to be able to provide the necessary power (some figures I've seen indicate that a modern nuclear reactor can generate up to 500 MW of power).<br />
<br />
There is some hope, however, if one takes into account accelerator architectures. These could, for various specialized types of tasks, dramatically increase the performance of each of their cores (some existing literature uses accelerator cores and regular CPU cores interchangeably). Energy consumption per core would also be affected if one uses these definitions. For example, there are many many GPU cores in a single GPU card. I do believe scientists in our field has already recognized these accelerator architectures as being essential to the success of exascale computing.<br />
<br />
Using the above assumptions about the hardware, what then needs to be considered for the software? Instead of looking at the general field of software in HPC, I would like to now focus on my sub-field of performance analysis of HPC applications. The very first thing that stares me in the face would be the core-counts. Traditionally, the performance data captured on each core conveniently represents the most accurate description of "what happened?". In the above scenario, we would now have to deal with information coming from 100 million cores. Do we keep 100 million files? Do we merge them? Do we try to summarize the information by various compression techniques (as I have explored in my Phd thesis)? The second challenge would be the massive number of events we would have to deal with. If my hypothesis about core speeds not climbing much faster is true, I would be grateful. For, if they did skyrocket, the same events would happen at a much higher frequency. That would in turn mean, if memory speeds fail to keep up with processor speeds (as they have traditionally failed to), recording these events would result in increased overheads and perturbation. Even if this did not happen, I would still have to deal with the sheer numbers for, in-lieu of speed, the application would be pumping the performance information out in parallel. The I/O subsystem had then better catch up if buffers get filled. Finally, what of the analysis? The new challenges I see, even if we do preserve all the details of performance data, is making sense of the increased heterogeneity of the events and the increased complexity of event interactions and how they affect performance. For example, communication between cores on the same chip should be vastly superior to communication between cores out-of-chip. The increased number of cores per chip could also make performance more sensitive to memory access, since the bandwidth between chip and memory gets tighter (given assumptions). The nature of GPGPU processing also adds new scenarios where performance may be sensitive to the way code is written, data shared or the datatypes used.<br />
<br />
This is exciting stuff and I look forward to dealing with these challenges as we march forward.Chee Wai Leehttp://www.blogger.com/profile/12854913855936196475noreply@blogger.com2tag:blogger.com,1999:blog-447827850717102908.post-25066663402112216052010-02-08T11:59:00.000-08:002012-07-22T12:27:20.971-07:00Dynamic Spanning Trees in HPCThis entry will be from the perspective of the Charm++ runtime and is intended to serve as a basis for wider discussion on the use of spanning trees, possibly dynamic, in other HPC software/middleware systems. As they stand, these are my personal thoughts and speculation. They are more questions than answers. Hopefully, as time goes by, they become more answers than questions.<br />
<br />
I assume the reader (myself included) understands the idea behind a spanning tree (and minimum spanning tree). In Charm++, we tend to construct spanning trees over the object communication graph in Charm++ applications. These spanning trees then serve as the backbone for broadcasts and reductions, where the root can be any vertex of the spanning tree.<br />
<br />
As I understand it, most other HPC runtimes use binary trees for their broadcast and reductions. If I am not wrong, this is because they do not need to deal with the dynamic nature of their applications. In Charm++, objects can migrate. As such, we have a dynamic runtime environment. Broadcasts and reductions happen in the background of application computation and other communication. I do not, however, know if spanning tree construction in Charm++ takes existing communication graph and volume into account. However, if broadcasts and reductions are to be efficient, they need to be woven around more static communication patterns. Hence, the construction of a minimum spanning tree, using the application's more static object communication volume as edge-weights.<br />
<br />
So, the questions, I suppose are:<br />
<br />
1. Are parallel spanning trees useful in general HPC software/middleware?<br />
<br />
2. Can our experience with spanning trees in Charm++ be directly applicable to other HPC systems that attempt to provide dynamic application behavior?Chee Wai Leehttp://www.blogger.com/profile/12854913855936196475noreply@blogger.com0