Trey's Webloghttp://longhorizon.org/blog/2013-06-05T16:02:28-07:00NASA Planetary Lake Lander Project at Google Tech Talk2013-06-05T16:02:28-07:00Trey Smithe2d3fa67-d339-40a5-56db-0064bb004a3a<p>Nathalie Cabrol gave a Google Tech Talk about our <a class="reference external" href="http://pll.seti.org/">Planetary Lake Lander
Project</a> project on Tuesday.</p>
<p>Titan is the only body in the solar system other than Earth that's known
to have lakes on its surface--only in Titan's case it's not water but
liquid methane and ethane at super-cold temperatures! NASA is studying
the possibility of landing a floating probe in one of these lakes and
our project uses a similar probe on Earth to learn how we will operate
the one on Titan.</p>
<p>Nathalie, our principal investigator, is a distinguished planetary
scientist, but also an amazing ball of energy who lights up everyone she
gets near. This was a fun talk.</p>
<iframe width="560" height="315" src="http://www.youtube.com/embed/n7ylyK1ycmg" frameborder="0" allowfullscreen></iframe><p>I'm leading the robotics part of the project. In past years we put the
probe together, and now we're developing the adaptive science component
that will allow the probe's onboard software to understand what its
science sensors are measuring and react appropriately, like increasing
the sampling rate during a storm. This is the most exciting phase of the
project and hopefully we'll be able to build some great new tools to
help the science team get better data.</p>
Joshua Tree Bloom in Mojave National Preserve2013-04-22T07:48:37-07:00Trey Smithe2d3fa67-d339-40a5-56db-0064bb004a3a<p>Sabrina and I recently visited <a class="reference external" href="http://www.nps.gov/moja/index.htm">Mojave National Preserve</a> to celebrate our wedding
anniversary. It was a quick weekend getaway and the timing was right to
see <a class="reference external" href="http://www.californiareport.org/archive/R201304190850/a">this year's record bloom</a> of <a class="reference external" href="http://en.wikipedia.org/wiki/Yucca_brevifolia">Joshua
trees</a>. I thought I
would share a few pictures. Click any photo to see a larger version.</p>
<div class="figure">
<a class="reference external image-reference" href="http://longhorizon.org/blog/static/images/2013_04_14_mojave/sabrina_joshua_bloom.jpg"><img alt="|filename|images/2013_04_14_mojave/sabrina_joshua_bloom_600.jpg" src="http://longhorizon.org/blog/static/images/2013_04_14_mojave/sabrina_joshua_bloom_600.jpg" style="width: 600px; height: 400px;" /></a>
<p class="caption">Sabrina enjoying a Joshua tree bloom</p>
</div>
<div class="figure">
<a class="reference external image-reference" href="http://longhorizon.org/blog/static/images/2013_04_14_mojave/lava_flow_aerial.jpg"><img alt="|filename|images/2013_04_14_mojave/lava_flow_aerial_600.jpg" src="http://longhorizon.org/blog/static/images/2013_04_14_mojave/lava_flow_aerial_600.jpg" style="width: 600px; height: 415px;" /></a>
<p class="caption">Cima Volcanic Field cinder cones and lava flow from the air</p>
</div>
<p>In February 2011, I was really lucky and happened to overfly <a class="reference external" href="http://digital-desert.com/cinder-cones/">Cima
Volcanic Field</a> while on a
commercial flight from Chicago to L.A. It's a set of 40 cinder cones and
associated lava flows 7.6 million years old, a part of the <a class="reference external" href="http://en.wikipedia.org/wiki/Basin_and_Range_Province">Basin and
Range geological province</a> in the
American Southwest.</p>
<p>I took some photos from the plane and used Google Earth to figure out
what I had been looking at when I got home. On this trip, I got to
visit some of the spots in my photo, and it was fun to make the
connection.</p>
<div class="figure">
<a class="reference external image-reference" href="http://longhorizon.org/blog/static/images/2013_04_14_mojave/lava_flow.jpg"><img alt="|filename|images/2013_04_14_mojave/lava_flow_600.jpg" src="http://longhorizon.org/blog/static/images/2013_04_14_mojave/lava_flow_600.jpg" style="width: 600px; height: 400px;" /></a>
<p class="caption">Cima Volcanic Field, Joshua tree and creosote bushes in foreground</p>
</div>
<div class="figure">
<a class="reference external image-reference" href="http://longhorizon.org/blog/static/images/2013_04_14_mojave/lava_flow_link.jpg"><img alt="|filename|images/2013_04_14_mojave/lava_flow_link_600.jpg" src="http://longhorizon.org/blog/static/images/2013_04_14_mojave/lava_flow_link_600.jpg" style="width: 600px; height: 200px;" /></a>
<p class="caption">Linking the aerial and ground photos of the volcanic field</p>
</div>
<p>The ground-level photo was taken from a wash that runs along the edge of
the lava flow. I marked the point where I took it from in the aerial
photo with a camera icon and a yellow shape showing the approximate
field of view.</p>
<p>The escarpment (cliff) running across the ground photo is the edge of
the lava flow, still looking steep and fresh after millions of
years. You can see two cinder cones marked A and B that appear in both
photos (the shape of cinder cone A is hard to make out in the aerial
photo due to the shadow of a passing cloud).</p>
<div class="figure">
<a class="reference external image-reference" href="http://longhorizon.org/blog/static/images/2013_04_14_mojave/beavertail_cactus_beetles.jpg"><img alt="|filename|images/2013_04_14_mojave/beavertail_cactus_beetles_600.jpg" src="http://longhorizon.org/blog/static/images/2013_04_14_mojave/beavertail_cactus_beetles_600.jpg" style="width: 600px; height: 400px;" /></a>
<p class="caption">Beavertail cactus bloom full of beetles</p>
</div>
<p>We saw several <a class="reference external" href="https://en.wikipedia.org/wiki/Opuntia_basilaris">beavertail cactus</a> blooms that were
swarming with little beetles. Maybe mating swarms of <a class="reference external" href="http://en.wikipedia.org/wiki/Blister_beetle">blister beetles</a>?</p>
<div class="figure">
<a class="reference external image-reference" href="http://longhorizon.org/blog/static/images/2013_04_14_mojave/lizard.jpg"><img alt="|filename|images/2013_04_14_mojave/lizard_600.jpg" src="http://longhorizon.org/blog/static/images/2013_04_14_mojave/lizard_600.jpg" style="width: 600px; height: 315px;" /></a>
<p class="caption"><a class="reference external" href="http://www.californiaherps.com/lizards/pages/u.scoparia.html">Mojave fringe-toed lizard</a> at <a class="reference external" href="http://en.wikipedia.org/wiki/Kelso_Dunes">Kelso Dunes</a></p>
</div>
<div class="figure">
<a class="reference external image-reference" href="http://longhorizon.org/blog/static/images/2013_04_14_mojave/solar_power.jpg"><img alt="|filename|images/2013_04_14_mojave/solar_power_600.jpg" src="http://longhorizon.org/blog/static/images/2013_04_14_mojave/solar_power_600.jpg" style="width: 600px; height: 356px;" /></a>
<p class="caption">Ivanpah Solar Project solar thermal energy tower and heliostats</p>
</div>
<p>Between our hotel in Primm, Nevada and the National Preserve, we drove
by some really crazy-looking architecture in the desert that looked kind
of like air traffic control towers without an airport. Eventually I
realized it must be a solar power plant. Turns out it was the <a class="reference external" href="http://en.wikipedia.org/wiki/Ivanpah_Solar_Power_Facility">Ivanpah
Solar Project</a>, the
largest solar thermal project under construction in the world.</p>
<p>In the photo you can actually see the beams of light from the heliostats
illuminating dust in the air and coming together in a focus like the
Death Star's planet-killing raygun. Interesting that they were mostly
not focused on the thermal energy tower, which seems not to be
operational yet. Anyway, pretty cool!</p>
<p>If you like these photos, you can also view <a class="reference external" href="https://plus.google.com/photos/100916908183085431209/albums/5869722533843197425">my trip album</a>.</p>
Ad Hoc Wireless Networking for Science in the Field2013-04-08T14:55:55-07:00Trey Smithc0b4e96d-0154-be0b-23b2-00d3d3004d16<p><a class="reference external" href="http://schwehr.org/blog/archives/2013-04.html#e2013-04-06T12_15_59.txt">Kurt recently posted some ideas for improving TrailScribe</a>.
In this post I want to discuss his first main idea: using an ad hoc
wireless network to transmit data between <a class="reference external" href="/blog/tag/trailscribe.html">TrailScribe units</a> in the field.</p>
<p>I think this is a great idea, so let's dig in and think about how you
could do that and how you would decide whether it's practical for a
given field deployment.</p>
<p>In current practice, a typical base camp in a remote area off the cell
network might have a BGAN satellite antenna (quick setup, ~ $10/MB
Internet connection for web browsing and email) and a single WiFi access
point that provides a data network within a few hundred feet of
camp. They'll also be using walkie-talkies to stay in touch during the
day.</p>
<div class="figure">
<a class="reference external image-reference" href="http://longhorizon.org/blog/static/images/2013_04_08_bcamp.jpg"><img alt="|filename|images/2013_04_08_bcamp_600.jpg" src="http://longhorizon.org/blog/static/images/2013_04_08_bcamp_600.jpg" style="width: 600px; height: 239px;" /></a>
<p class="caption">Typical base camp network</p>
</div>
<p>Suppose we'd like to extend the data network to cover the whole field
operation so people can get updates on their TrailScribes. Partly they
want to share data with each other and partly they want to get updates
from the outside world through the satellite connection (for example,
check the latest weather forecast during the day).</p>
<p>Ideally, any changes we propose should piggyback on the time and money
our users have already invested in their equipment, and not create more
work for them in the field.</p>
<p>Given that context, let's explore a few natural approaches to extending
the data network outside camp.</p>
<div class="section" id="wifi-ad-hoc-networking">
<h2>WiFi Ad Hoc Networking</h2>
<p>The base camp network uses WiFi and our users are familiar with it. Why
not use it for the extended network? This is a pretty good approach.</p>
<p>Companies like <a class="reference external" href="http://www.tropos.com/products/performance_mesh.html">Tropos</a> provide
off-the-shelf WiFi ad hoc networking gear. If you get multiple WiFi
repeaters you can spread them out in whatever pattern you like and
clients at any point in the network can send data to each other. The
repeaters detect link quality and configure multi-hop routes
automatically.</p>
<div class="figure">
<a class="reference external image-reference" href="http://longhorizon.org/blog/static/images/2013_04_08_wifi.jpg"><img alt="|filename|images/2013_04_08_wifi_600.jpg" src="http://longhorizon.org/blog/static/images/2013_04_08_wifi_600.jpg" style="width: 600px; height: 276px;" /></a>
<p class="caption">User 1 can send data to user 2 with multiple hops</p>
</div>
<p>There are two main challenges here. The first is that even off-the-shelf
ad hoc networking equipment requires a reasonably experienced network
admin in the field, and we find that physical setup (transceiver +
battery + tripod mount) and debugging network problems can take a lot of
time.</p>
<p>The second is coverage. Because WiFi equipment operates in unlicensed
spectrum, the U.S. FCC and similar agencies elsewhere put fairly strict
limits on how much power you can use, to avoid interference with other
users. The maximum range is pretty short. If you need to cover a lot of
ground you need a lot of repeaters, and the problem gets worse if
there's interesting terrain in the way because repeaters need line of
sight to talk.</p>
<p>Another thing to think about is the last-mile issue. Modern tablets have
built-in WiFi, but with a low-power transceiver and low-gain antenna
that reduces their range. To reduce the number of fixed repeaters you
need, users may end up carrying a backpackable repeater with a bit more
range.</p>
</div>
<div class="section" id="vhf-hacking">
<h2>VHF Hacking</h2>
<p>This is an area I know less about, and I would love to hear more from
experts.</p>
<p>VHF radios operate at lower frequencies than WiFi and typically use
parts of the spectrum where the team needs to get a license but then is
allowed to use more power. VHF comm links sometimes work in
near-line-of-sight conditions where one of the radios is out of sight of
the other one (slightly over the horizon). What all this boils down to
is VHF usually provides less bandwidth but you can cover more ground
with a single repeater.</p>
<p>Many field teams use VHF walkie-talkies, and they may already be
planning to cover the operational area with VHF repeaters to carry voice
traffic. Can we piggyback small amounts of data on this network? What
problems do we need to think about?</p>
<p>First, we need to check that data transmission is not prohibited on the
frequency the team is using. That will depend on the conditions in the
license and I'm not sure how common a problem it is.</p>
<p>Next, we need to figure out how to connect a tablet to the VHF network
in the field. In an ideal world, we could pair the tablet and the user's
walkie-talkie with a wireless connection (like WiFi or bluetooth). That
would allow users to use both the tablet and the walkie-talkie with no
encumbering wires and no extra radio equipment. I'm not currently aware
of any walkie-talkies that can do that, but that could change any day
now. <a class="reference external" href="http://deusexmachina.org/">(Tell Mark it would be a good weekend project.)</a></p>
<p>A compromise approach I think might work ok is putting together a small
backpackable unit that includes a <a class="reference external" href="https://github.com/ytai/ioio/wiki/IOIO-Over-Bluetooth">IOIO board with a bluetooth dongle</a> to pair
wirelessly with the tablet, and a <a class="reference external" href="http://www.byonics.com/tinytrak/">TNC</a> that connects it to the audio port
of a dedicated VHF transceiver. The user would still need to carry a
separate VHF handheld, but no need for wires running to the tablet.</p>
<div class="figure">
<a class="reference external image-reference" href="http://longhorizon.org/blog/static/images/2013_04_08_vhf.jpg"><img alt="|filename|images/2013_04_08_vhf_600.jpg" src="http://longhorizon.org/blog/static/images/2013_04_08_vhf_600.jpg" style="width: 600px; height: 323px;" /></a>
<p class="caption">Small backpackable unit for bridging a tablet to a VHF network</p>
</div>
<p>Beyond the pairing problem, data going over the VHF network is going to
be pretty slow and may need to share the channel with voice traffic, so
we're probably in the realm of passing short messages rather than
viewing media-rich web pages, and our applications need to be designed
accordingly.</p>
</div>
<div class="section" id="how-i-learned-to-stop-worrying-and-love-dedicated-hardware">
<h2>How I Learned To Stop Worrying and Love Dedicated Hardware</h2>
<p>Coming from a computer science background, I find the idea of dedicated
comms hardware (like voice-only radios) really painful. It seems
blindingly obvious that we should have a general-purpose computer
attached to the radio, it should be able to send arbitrary bits down the
pipe, and applications like voice comms should be managed by software
you can change.</p>
<p>If you're trying to do novel research by writing brand-new apps,
figuring out general-purpose data comms is important. But if you just
want to get the job done, dedicated hardware still wins pretty often at
the moment.</p>
<p>Here's one example: If all you really need is position tracking, you can
use a dedicated unit like the <a class="reference external" href="http://www.amazon.com/gp/product/B000065DQ2/ref=as_li_tf_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=B000065DQ2&linkCode=as2&tag=tresweb-20">Garmin Rino</a>. Rinos
are combined GPS / walkie-talkies that exchange position updates so you
can see where your team-mates are. We've had great luck with these on
the <a class="reference external" href="http://pll.seti.org/">Planetary Lake Lander</a> project, and we've
started looking into ways to better integrate them with other data, like
<a class="reference external" href="http://www.xastir.org/wiki/Configuring_Xastir#Enabling_Garmin_RINO_Support">using a base station to log position updates</a>.</p>
<p>Rinos have all kinds of problems--we can't change how they work, they
can't send position updates over a repeater, and their tiny map screens
leave a lot to be desired. But they can replace a GPS and a
walkie-talkie, they're cheap, small, and rugged, and they don't need
expert admins. It will take a lot of work before we can match that kind
of ease of use with a tablet-based solution.</p>
</div>
Digital Field Assistant: The Context Around TrailScribe2013-04-07T18:00:18-07:00Trey Smithb1408406-a61b-5e9f-b759-0090ac000b55<p>Yesterday <a class="reference external" href="http://schwehr.org/blog/archives/2013-04.html#e2013-04-06T12_15_59.txt">Kurt posted some ideas for improving TrailScribe</a>.
I really like the ideas and I may talk more about them in future posts.</p>
<p>They also reminded me of similar themes from my "Digital Field
Assistant" concept that <a class="reference external" href="/blog/tag/trailscribe.html">TrailScribe</a> grew out of, and I
realized I never shared my write-ups on that from back in 2011.</p>
<p>So to complement the discussion... here's an old presentation that
basically frames the core (TrailScribe) mobile device as part of a
broader "field data system" that includes helmet cams, specialized
sensors like Bluetooth weather gauges, VHF handheld radios, and an
information sharing server in base camp.</p>
<p>Click the gear and "Open speaker notes" to get the full content.</p>
<iframe src="https://docs.google.com/presentation/d/1equDpzp58252w_jMaWSYlbTmfcG3rHqyeOgru3rEVOo/embed?start=false&loop=false&delayms=3000" frameborder="0" width="528" height="437" allowfullscreen="true" mozallowfullscreen="true" webkitallowfullscreen="true"></iframe><p>You can think of the <a class="reference external" href="/blog/2013/02/26/trailscribe-tablet-for-field-science/">TrailScribe concept video</a>
as a down-sized version of this initial vision, focusing on the
low-hanging fruit: What can you do if all you can afford is some
tablet computers and cheap accessories like bluetooth headsets and
ruggedized cases?</p>
Improving Python Development in Emacs with YASnippet2013-03-31T10:31:50-07:00Trey Smithda937674-9a28-11e2-9608-77c06318632b<p><a class="reference external" href="http://schwehr.org/">Kurt Schwehr</a> just showed me how he uses the
<a class="reference external" href="https://github.com/capitaomorte/yasnippet#readme">YASnippet</a> template
system to improve his productivity in Emacs. This seems like a really
great tool.</p>
<p>Here's an example. With YASnippet installed, if you're editing a Python
file in Emacs and you type the word <tt class="docutils literal">class</tt> then hit <tt class="docutils literal">[TAB]</tt>, it will
expand to the skeleton of a Python class definition:</p>
<div class="highlight"><pre><span class="k">class</span> <span class="nc">ClassName</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> """</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> """</span>
</pre></div>
<p>That's not all. After expansion, your cursor is left sitting on
<tt class="docutils literal">ClassName</tt> so you can immediately edit the class name to what you
want, and each time you hit <tt class="docutils literal">[TAB]</tt> after that, it will advance the
cursor to logical edit points, skipping over the boilerplate pieces you
wouldn't want to change. Some snippets also call out to other Emacs
functions to do things like auto-fill a current timestamp.</p>
<p>YASnippet is not just for Python. You can define mode-specific snippets
for any mode. YASnippet ships with a few pre-defined snippets for
several modes, including the <tt class="docutils literal">class</tt> snippet for Python mode that we
just used.</p>
<p>If you're interested in using YASnippet, here's the basic process for
setting it up:</p>
<blockquote>
<ul class="simple">
<li>Install YASnippet</li>
<li>Try out its pre-defined snippets in your favorite Emacs mode</li>
<li>Look around the web for additional snippet collections specific to your needs</li>
<li>Write your own snippets</li>
</ul>
</blockquote>
<p>Let's go through the specifics of setting up YASnippet for Python/Django
development. First grab the library.</p>
<div class="highlight"><pre><span class="nb">cd</span> ~/.emacs.d/plugins
git clone https://github.com/capitaomorte/yasnippet.git
</pre></div>
<p>Then add this to your <tt class="docutils literal"><span class="pre">~/.emacs</span></tt> init file.</p>
<div class="highlight"><pre>(add-to-list 'load-path
"~/.emacs.d/plugins/yasnippet")
(require 'yasnippet)
(setq yas-snippet-dirs
'("~/.emacs.d/plugins/yasnippet/snippets"))
(yas-global-mode 1)
</pre></div>
<p>(Note: You may want to compare this with the latest version of the
YASnippet README in case the init file setup directions have changed. I
had trouble at first due to version skew with old init file examples.)</p>
<p>At this point you can fire up Emacs and try out the <tt class="docutils literal">class</tt>
tab-completion example above. Now let's add an additional snippet
collection for Python/Django development.</p>
<div class="highlight"><pre><span class="nb">cd</span> ~/lib/elisp
git clone https://github.com/gabrielelanaro/emacs-for-python.git
</pre></div>
<p>The <tt class="docutils literal"><span class="pre">emacs-for-python</span></tt> package is an enormous kitchen sink of every
possible Python/Emacs tool you could think of. I've barely scratched the
surface of what's in there. But for now we're just interested in its
snippet collection. To tell YASnippet how to find the new collection,
edit your init file:</p>
<div class="highlight"><pre>(setq yas-snippet-dirs
'("~/lib/elisp/emacs-for-python/snippets/django"
"~/.emacs.d/plugins/yasnippet/snippets")
</pre></div>
<p>This gives us a bunch of snippets for Django, like <tt class="docutils literal">model</tt> in Python
mode and <tt class="docutils literal">block</tt> in HTML mode (for editing Django templates).</p>
<p>Finally, let's set up a trivial snippet of our own just to see how it's
done. We'll keep our own personal snippet collection in the suggested
location <tt class="docutils literal"><span class="pre">~/.emacs.d/snippets</span></tt>. Edit your init file:</p>
<div class="highlight"><pre>(setq yas-snippet-dirs
'("~/.emacs.d/snippets"
"~/lib/elisp/emacs-for-python/snippets/django"
"~/.emacs.d/plugins/yasnippet/snippets")
</pre></div>
<p>Notice our personal snippets area is first in the list. Order is
significant--this way you can override a built-in snippet by defining a
personal snippet with the same name.</p>
<p>Our trivial example snippet will expand <tt class="docutils literal">foo</tt> into <tt class="docutils literal">foobar</tt> in Python mode.
To set it up, create the file <tt class="docutils literal"><span class="pre">~/.emacs.d/snippets/python-mode/foo</span></tt> with these
contents:</p>
<div class="highlight"><pre><span class="c"># -*- mode: snippet -*-</span>
<span class="c"># name: foo</span>
<span class="c"># key: foo</span>
<span class="c"># --</span>
foobar
</pre></div>
<p>To test it out, restart Emacs, edit a Python file, type <tt class="docutils literal">foo</tt> and hit
<tt class="docutils literal">[TAB]</tt>. You should be good to go. If you'd like to make more
interesting snippets, you can model them on examples from one of the
existing snippet collections. Have fun!</p>
Best Talks at PyCon US 20132013-03-24T16:00:19-07:00Trey Smith95628a38-94d6-11e2-95c7-fb91dbb3cff8<p>I didn't make it to PyCon this year but I'm really enjoying the <a class="reference external" href="http://pyvideo.org/category/33/pycon-us-2013">talk
videos posted at PyVideo</a>. If you're interested
in <a class="reference external" href="http://wiki.python.org/moin/BeginnersGuide/Overview">Python</a>,
these are very worth watching. The production quality is good and they
are under 30 minutes each. Here are some of my favorites.</p>
<p>[<a class="reference external" href="http://pyvideo.org/video/1738/good-enough-is-good-enough">Alex Martelli: "Good enough" is good enough!</a>] For me,
watching this talk is like going to a revival meeting, getting fired up
all over again, then having this uncomfortable realization that you have
a lot to live up to.</p>
<iframe width="420" height="315" src="http://www.youtube.com/embed/yo4Uqq7NXQc" frameborder="0" allowfullscreen></iframe><p>[<a class="reference external" href="http://pyvideo.org/video/1742/how-not-to-build-an-oss-community">Daniel Lindsley: How (Not) To Build An OSS Community.</a>]
This is another talk I find humbling. Our research group at NASA builds
a lot of open source software that works well for the needs of our
clients, but we have never had a lot of success engaging a wider user
base or outside developers. Daniel does a great job of laying out what
we need to do, but are not doing.</p>
<iframe width="420" height="315" src="http://www.youtube.com/embed/o0jY0_OEyRc" frameborder="0" allowfullscreen></iframe><p>[<a class="reference external" href="http://pyvideo.org/video/1727/solid-python-application-deployments-for-everybod">Hynek Schlawack: Solid Python Application Deployments For Everybody.</a>]
Watch if you deploy web apps. Hugely entertaining. You may agree or
disagree but you will never be left wondering what Hynek thinks! <a class="reference external" href="http://hynek.me/talks/python-deployments/">His
speaker slides have the
links.</a></p>
<iframe width="420" height="315" src="http://www.youtube.com/embed/voq4Mm58NY0" frameborder="0" allowfullscreen></iframe>NASA xGDS Analysis Notebook Built on iPython Notebook2013-03-15T17:09:09-07:00Trey Smithb45152cc-8dcd-11e2-82ae-43877f0b12d3<p>The <a class="reference external" href="http://xgds.org">NASA xGDS</a> Analysis Notebook is a new tool I'm
developing that I'm really excited about. It's built on the iPython
Notebook <a class="footnote-reference" href="#id2" id="id1">[1]</a>, an interactive MATLAB-like shell within a web browser
that lets you do all kinds of numerical analysis and data visualization
with just a few lines of Python code. I'm extending it with hooks to
our live database and to simple customized plotting functions so our
science teams can get right into analyzing their data.</p>
<p>The way the notebook works is pretty simple. It's split up into
cells. You enter a command into a cell and it immediately displays the
result below, then gives you a new cell.</p>
<p>Here's an example using <a class="reference external" href="http://www.nasa.gov/exploration/systems/ground/resolverover.html">NASA RESOLVE rover</a>
data where I quickly wrote a plotting function that queries records from
two different database tables (neutron spectrometer data and position
data), resamples both data sets to the same time frequency, aligns them
by timestamp, and plots the result as a colored trail on a map. Even if
you're not familiar with the functions we're calling, you can see these
tools let us work quickly at a high level without sweating the details.</p>
<div class="figure">
<a class="reference external image-reference" href="http://longhorizon.org/blog/static/images/2013_03_14_nsRoverTrail.png"><img alt="RESOLVE neutron spectrometer signal plotted as a colored rover trail on a map." src="http://longhorizon.org/blog/static/images/2013_03_14_nsRoverTrail_600.jpg" style="width: 600px; height: 777px;" /></a>
<p class="caption">RESOLVE neutron spectrometer signal plotted as a colored rover trail on a map.</p>
</div>
<p>I'll have a lot to say about the analysis notebook going forward, but I
thought I would start by putting it in context with the rest of the xGDS
project.</p>
<p>xGDS develops software to support science operations, ranging from
pre-mission planning to in-mission monitoring to post-mission data
analysis. Until now, the data analysis part has been pretty thin. We
had pre-configured displays that let you plot and map and search data,
but no way to really process it.</p>
<p>The analysis notebook tool will give us a lot more depth in that
area. You get a very flexible numerical toolkit and you can look for
unexpected correlations between different data sets, check for agreement
between measurements and numerical models, whatever you want.</p>
<p>We can compare the analysis notebook to our old approach of providing
bulk data export to the science team in simple formats like CSV files,
so they can import and analyze the data in familiar tools like Excel or
MATLAB or IDL, and use specialized analysis tools developed for their
discipline (like spectroscopy). We will absolutely continue supporting
that and we believe it's vital, but it shouldn't be the whole
story. Using the analysis notebook has some important advantages:</p>
<blockquote>
<ul class="simple">
<li><strong>Sharing.</strong> Sharing is on by default. If you make a new plot,
everyone on the science team can see it right away, and if you update
it they always see the latest version.</li>
<li><strong>Accessibility.</strong> You don't have to install expensive specialized
commercial software to use the analysis notebook. Or, in fact, <em>any</em>
software, except a web browser. That also means it's not restricted
to any particular operating system. Cross-disciplinary collaboration
is easier if the different disciplines can use the same tools.</li>
<li><strong>Self-documenting.</strong> Using the notebook encourages you to document
your work processes. Anyone who can read your notebook sees not just
your results, but also the techniques you used to generate them, so
we can all improve our process by example. At its best, a notebook is
not just a plot but a kind of scientific essay about what you're
trying to understand and why you needed to process the data in the
way you did. This relates to some important modern progress in the
philosophy of science, as embodied in the <a class="reference external" href="http://www.rrplanet.com/">reproducible research</a> and <a class="reference external" href="http://en.wikipedia.org/wiki/Literate_programming">literate programming</a> movements.</li>
<li><strong>No export/import overhead.</strong> The notebook environment has "native"
access to the live database, reducing the overhead of exporting data
then importing into a new tool.</li>
<li><strong>Science/engineering collaboration.</strong> Especially important to us on
the xGDS team, the notebook can be a way for us to work more closely
with the science team. The scientists are experts on their
instruments and on data interpretation, whereas we are experts in
numerical data analysis and how the database is organized. Using the
notebook collaborative environment, we should be able to do better
data analysis together. (Plus, scientists are cool.)</li>
</ul>
</blockquote>
<p>Beyond post-mission data analysis, the other place the analysis notebook
can help is as a prototyping tool for live console displays. We can
work with a science team to try out lots of visualizations, pick out our
favorites, then package them up for operational use as console displays
that can update based on real-time data and handle the load from lots of
users.</p>
<p>I also got some good initial feedback--this week I showed the analysis
notebook to <a class="reference external" href="http://www.nasa.gov/centers/ames/research/2007/colaprete.html">Tony Colaprete</a> and <a class="reference external" href="http://www.youtube.com/watch?v=mnkG-8YGUJI">Rick
Elphic</a>. Rick's top
comment was that we need to connect the notebook to Google Earth so we
can plot things in context with other map data. C'mon, Rick, don't throw
me in that briar patch!</p>
<table class="docutils footnote" frame="void" id="id2" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>There's actually a whole stack of libraries we're building on:
<a class="reference external" href="http://nbviewer.ipython.org/">iPython Notebook</a> for the web-based
shell, <a class="reference external" href="http://pandas.pydata.org/">pandas</a> for structured data
analysis, <a class="reference external" href="http://matplotlib.org/">matplotlib</a> for plotting, and
<a class="reference external" href="http://www.scipy.org/">SciPy</a> and <a class="reference external" href="http://www.numpy.org/">NumPy</a>
for general numerical analysis.</td></tr>
</tbody>
</table>
NASA Exploration Ground Data System (xGDS) at Google Tech Talk2013-03-15T16:53:50-07:00Trey Smith8f9faae8-8dcb-11e2-b9e8-c74ada26a0cc<p>Matt Deans gave a Google Tech Talk about our <a class="reference external" href="http://xgds.org/">NASA Exploration Ground
Data Systems (xGDS)</a> project on Tuesday. Matt is the
xGDS project lead and I think this is the best overview presentation
I've seen yet.</p>
<iframe width="560" height="315" src="http://www.youtube.com/embed/CrDdtVEJyCk" frameborder="0" allowfullscreen></iframe><p>To support the talk I finally got around to making an <a class="reference external" href="http://xgds.org/">xGDS project web
page</a>. We've done so many interesting interfaces and
supported so many cool science teams that the page really doesn't do the
project justice, but it's a start...</p>
Migrating a repository from CVS and Subversion to Git with history2013-03-03T21:39:23-08:00Trey Smith0de7666e-8473-11e2-8912-4fdee2295da0<p>Today I migrated my old <a class="reference external" href="http://github.com/trey0/zmdp">ZMDP planning software</a> from cvs and svn repositories to git. I
got pretty deep into some undocumented stuff, so here are my notes.</p>
<p>My basic plan was to migrate from cvs to svn using <a class="reference external" href="http://cvs2svn.tigris.org/">cvs2svn</a>, then migrate from svn to git using
<a class="reference external" href="https://www.kernel.org/pub/software/scm/git/docs/git-svn.html">git-svn</a>. But
there were some other problems I also needed to fix.</p>
<p>First install the tools. I'm using <a class="reference external" href="http://www.macports.org/">MacPorts</a> <a class="footnote-reference" href="#id2" id="id1">[1]</a>.</p>
<div class="highlight"><pre><span class="nv">$ </span>sudo port -vn install cvs2svn
<span class="c"># git-core does not include git-svn by default</span>
<span class="nv">$ </span>sudo port -vn install git-core +svn
</pre></div>
<p>Then migrate the cvs repo to svn.</p>
<div class="highlight"><pre><span class="nv">$ </span>cp -a ~/projects/zmdp/repository cvsrepo
<span class="nv">$ </span>cvs2svn -s svnrepo cvsrepo
</pre></div>
<p>Notice I made a backup of the cvs repository first in case <tt class="docutils literal">cvs2svn</tt>
did something bad. But no problem, that worked great and preserved the
cvs history.</p>
<p>Here's where my first problem comes in. Several years ago I needed to
migrate this codebase from cvs to svn in a hurry and I did it the quick
and dirty way, by creating a fresh svn repository from a cvs checkout,
which lost the cvs history. Since then I made about ten commits in
the old svn repo, and now I want to tack those commits onto the end of
the cvs history that I migrated into the new svn repo.</p>
<p>Long story short, it seems to be easy to merge the <em>content</em> of two
branches in svn but hard to merge their commit history. The
straightforward <tt class="docutils literal">svn merge</tt> operation basically squashes the commit
history of whatever you merge in together into one commit.</p>
<p>Time for a work-around. Apply each revision of the old svn repo
individually to the new repo, commit it with the correct comment, then
set the <tt class="docutils literal">svn:date</tt> property of the revision to match the original
commit time.</p>
<p>But first I had to set the <tt class="docutils literal">svn</tt> settings to allow editing revision
properties. Apparently <tt class="docutils literal">svn</tt> considers that kind of thing suspicious
by default. In order to allow it you have to make the
<a class="reference external" href="http://svnbook.red-bean.com/en/1.7/svn.ref.reposhooks.pre-revprop-change.html">pre-revprop-change</a>
hook a valid executable that exits with a successful status
of 0. The web is short on examples of how to do this.</p>
<div class="highlight"><pre><span class="nv">$ </span><span class="nb">cd</span> ~/sandbox/cvs2svn/svnrepo/hooks
<span class="nv">$ </span>cat <<<span class="s2">"EOF"</span> > pre-revprop-change
<span class="c">#!/bin/bash</span>
<span class="nb">exit </span>0
EOF
<span class="nv">$ </span>chmod +x pre-revprop-change
</pre></div>
<p>Now apply, commit, and edit the old commits one at a time. If there were
more than ten I'd have to script it, but this was just barely ok to do
manually.</p>
<div class="highlight"><pre><span class="nv">$ </span><span class="nb">cd</span> ~/projects/zmdp/svnRepository/src
<span class="nv">$ </span>svn log
------------------------------------------------------------------------
r10 | mfsmith3 | 2010-08-16 14:48:04 -0700 <span class="o">(</span>Mon, 16 Aug 2010<span class="o">)</span> | 1 line
updated requirements section of README, new tested compilers
...
<span class="nv">$ </span><span class="nb">cd</span> /tmp/newsvncheckout
<span class="nv">$ </span>svn merge -r 9:10 file:///Users/mfsmith3/projects/zmdp/svnRepository/src
<span class="nv">$ </span>svn commit -m <span class="s1">'updated requirements section of README, new tested compilers'</span>
<span class="nv">$ </span>svn propset svn:date <span class="s1">'2010-08-16T21:48:04.0Z'</span> --revprop -r HEAD <span class="se">\</span>
file:///Users/mfsmith3/sandbox/cvs2svn/svnrepo
</pre></div>
<p>Now transition everything to git.</p>
<div class="highlight"><pre><span class="nv">$ </span>git svn clone file:///Users/mfsmith3/sandbox/cvs2svn/svnrepo/trunk/src
</pre></div>
<p>Things were looking good, but I made a last pass through the git commit
log and noticed that the authorship was messed up--I prefer a clean name
and email address in the git log but these commits just had an old UNIX
username. Luckily, git has much better tools for <a class="reference external" href="http://git-scm.com/book/ch6-4.html">editing the commit
history</a> than svn does.</p>
<div class="highlight"><pre><span class="nv">$ </span><span class="nb">echo</span> > filter.sh <<<span class="s2">"EOF"</span>
<span class="c">#!/bin/sh</span>
git filter-branch --commit-filter <span class="s1">'</span>
<span class="s1"> if [ "$GIT_AUTHOR_NAME" = "trey" ];</span>
<span class="s1"> then</span>
<span class="s1"> GIT_AUTHOR_NAME="Trey Smith";</span>
<span class="s1"> GIT_AUTHOR_EMAIL="trey.smith@gmail.com";</span>
<span class="s1"> git commit-tree "$@";</span>
<span class="s1"> else</span>
<span class="s1"> git commit-tree "$@";</span>
<span class="s1"> fi'</span> HEAD
EOF
<span class="nv">$ </span>./filter.sh
</pre></div>
<p>All done...</p>
<table class="docutils footnote" frame="void" id="id2" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>The <tt class="docutils literal"><span class="pre">-n</span></tt> option to <tt class="docutils literal">port</tt> is one I almost always
use--it keeps <tt class="docutils literal">port</tt> from aggressively trying to upgrade everything
the package you're installing depends on. There's nothing like
installing a minor package through <tt class="docutils literal">port</tt> and having it download
and recompile a dozen other packages including new Python and Perl
interpreters because somebody bumped the patchlevel. If you're not
careful, those upgrades can also orphan and break other packages. If
you want to stay up to date you're probably better off running <tt class="docutils literal">port
upgrade outdated</tt> on a weekly basis.</td></tr>
</tbody>
</table>
Making remote software updates more reliable by rotating versions2013-03-01T13:19:07-08:00Trey Smitha1c52ee4-82b5-11e2-9c6b-b7f51dfb8f20<p>How do we ensure our exploration robots keep working after software
updates?</p>
<p>Traditionally, space missions solve this problem by:</p>
<blockquote>
<ul class="simple">
<li>Avoiding advanced software techniques that are seen as risky. Once
upon a time, "advanced software techniques" included compilers.
These days, the team might enforce coding conventions like not
throwing exceptions, so you have to write explicit handlers to
recover from every fault.</li>
<li>Thorough code review by the mission's software engineering
team. Changes are vetted by a change control board that decides
whether the new functionality is worth the risk of making a change
that could break the system.</li>
<li>Automated software verification. This is everything from <a class="reference external" href="http://en.wikipedia.org/wiki/Lint_%28software%29">lint</a> to
special-purpose tools that will do things like <a class="reference external" href="http://en.wikipedia.org/wiki/Java_Pathfinder">detect possible race
conditions in multi-threaded software</a>.</li>
<li>Testing in simulation. Sometimes these simulations are very
elaborate. It could be a video game-like simulation of a detailed
planetary surface environment for testing rover software, or a
simulation of a system of valves and motors that injects faults like
jammed valves or bad sensor data.</li>
<li>Building a twin of the flight hardware that stays behind on the
ground to support testing software updates in a realistic hardware
environment.</li>
</ul>
</blockquote>
<p>These are all great techniques, but sometimes you don't have the
resources to do all the testing you want. That's the situation we're
facing on the <a class="reference external" href="http://pll.seti.org">NASA Planetary Lake Lander Project</a>. Our robotic probe is a buoy sitting in a
high-altitude lake near Santiago, Chile. It collects water quality,
weather, and health data and beams daily updates back to us via a BGAN
satellite link.</p>
<p>The lake is pretty remote. During the Southern Hemisphere summer we can
send an engineer to service the probe about once a month, but during the
winter the lake is snowed in. If we lose contact with the probe we could
end up losing months' worth of data.</p>
<p>One of PLL's research goals is to use the probe to conduct adaptive
science, where it intelligently chooses what measurements to take and
what data to beam back based on scientific relevance. Ideally, we'll
build the adaptive science system over the course of this year and
deploy incremental updates to the probe as we bring new features online.
But if any of the updates break our core software services that schedule
activities and connect to the satellite network, there would be no way
for us to remotely contact the probe to fix it.</p>
<div class="section" id="rotating-versions">
<h2>Rotating Versions</h2>
<p>We've hit on a simple technique we hope will help recover from that
problem. When we install a new version of the software, we'll keep older
versions around as well. Instead of totally switching over to the new
version, we actually set up a rotation where we switch back and forth
between versions every few days. That way, if the new version is broken
and doesn't allow us to make contact with the probe, eventually the old
version will rotate back in and we'll regain contact. On the other hand,
if the new version works out really well, we can manually step in to
disable the rotation before it switches back.</p>
<p>Some key considerations for this strategy:</p>
<blockquote>
<ul class="simple">
<li>First, do no harm. The world is full of "safety systems" that
actually make things less reliable. We put lots of sanity checks into
our rotation software, and if it finds any surprises, it gives up and
doesn't change anything.</li>
<li>Rotation makes the most sense if you have a critical need (eventually
make contact), so you want to try everything, but none of the options
is likely to cause permanent damage to the system. Luckily, our probe
is moored in place on the lake, so it's hard to get into trouble.</li>
<li>There is no silver bullet. Even with rotation in place we plan to
carefully verify each new software version to the extent that we can.</li>
<li>It's important to keep the software that does the version rotations
simple and independent from the rest of the software. Don't rotate
the rotation script!</li>
</ul>
</blockquote>
<p>Time will tell if this is a good strategy. It would be interesting to
hear who else has hit on the same idea.</p>
</div>
TrailScribe tablet for field science2013-02-26T16:44:09-08:00Trey Smith2285ccac-605a-4ecc-8e94-004080adce4c<p>TrailScribe is a concept design for how scientists could use tablet
computers as enhanced field notebooks to improve their workflow.</p>
<iframe width="420" height="315" src="http://www.youtube.com/embed/xVMNhJy8lCY"
frameborder="0" allowfullscreen></iframe><p>TrailScribe is one piece of the "Digital Field Assistant" concept I
first started talking about in 2011. The idea is to organize field
science data collection and sharing with a combination of phones,
tablets, sensors, laptop servers, and the software to make them work
together.</p>
<p><a class="reference external" href="http://www.linkedin.com/pub/young-wook-jung/56/704/327">Youngwook Jung</a>, a very
talented Ph.D. design student from <a class="reference external" href="http://www.kaist.edu/edu.html">KAIST</a>, worked with me in Fall 2012 to
develop interface wireframes and artwork and produce this video.</p>
<p>TrailScribe tablets have three main uses:</p>
<blockquote>
<ul class="simple">
<li><strong>Data collection.</strong> A scientist collecting a data set in the field
can define the format of the data records they want to collect,
including timestamps, GPS position, photos, bar codes identifying
sample bags, and voice or touch data entry. TrailScribe can fill
some of these fields automatically and simplifies others. The
objective is to reduce data collection effort and produce a richer
and better organized data set.</li>
<li><strong>Situation awareness.</strong> With overnight synchronization through a
laptop server in the field, all members of the science team get
up-to-date access to all the data collected so far, even while they
are offline in the field.</li>
<li><strong>Field safety.</strong> By daily sharing of digital traverse plans and
tracking check-ins with an automated roster, the team can keep better
tabs on each other, improving field safety.</li>
</ul>
</blockquote>
<p>Young and I had so many ideas for TrailScribe that it was a real
challenge to focus on just a few so we could produce a short video. One
guiding principle was to look for low-hanging fruit: it's easier to
change workflow than develop software, and it's easier to develop
software than to develop hardware. All of the functionalities
illustrated in the concept video can be implemented with off-the-shelf
hardware components.</p>
<p>Another way of looking at TrailScribe is that it merges the field test
support features we developed on the Exploration Ground Data Systems
Project with our experience supporting field operations using mobile
apps on the <a class="reference external" href="http://geocamshare.org/">NASA GeoCam Project</a>. This
background puts us in a great position to implement the TrailScribe
system.</p>
<p>Going forward, we're looking for a field science operation to
collaborate with on a funding proposal, so we can take this past the
concept design stage.</p>
MapFasten map alignment web app2013-02-25T19:59:04-08:00Trey Smithaf689cdb-bee6-44a0-85b7-5541a574de32<p>MapFasten (<a class="reference external" href="http://mapfasten.appspot.com/">web app</a>, <a class="reference external" href="https://github.com/geocam/geocamMapFasten/">source on GitHub</a>) is a tool for
aligning an image or PDF with other map layers so you can use it more
effectively.</p>
<iframe width="560" height="315" src="http://www.youtube.com/embed/88gt1gj2dbs"
frameborder="0" allowfullscreen></iframe><p>The basic idea is that you find matching landmarks in your overlay and
in the underlying map, add tie points to those landmarks, then the
MapFasten solver will find the best alignment that makes the matching
tie points line up. From there you can share the aligned overlay with
your team and view it in other maps.</p>
<p>Our <a class="reference external" href="http://geocamshare.org">NASA GeoCam Project</a> team built MapFasten
and our main clients were the <a class="reference external" href="http://www.google.org/crisisresponse">Google Crisis Response Team</a>.</p>
<p>After a major disaster, the Crisis Response Team generates a Crisis Map,
which is a set of many map layers that provide different kinds of
information to aid responders and citizens in the affected area.
Local organizations often have data that's important to share through a
Crisis Map, but sometimes it comes in the form of a PDF or a scan of a
printed document, so it's hard to combine with other map layers.</p>
<p>The goal of MapFasten is to make it easier for those organizations to
import their images, align them, and publish them in the Google Maps
tiles format, which makes it easy to import the data into a Crisis
Map. Although MapFasten's initial purpose is to support crisis response,
it can be used to align any kind of map image and the result can be
easily added to any map built on the Google Maps API.</p>
<div class="figure">
<a class="reference external image-reference" href="http://longhorizon.org/blog/static/images/2012_08_hurricaneIsaacCrisisMapScreenShot.png"><img alt="Map of New Orleans levee system aligned by MapFasten" src="http://longhorizon.org/blog/static/images/2012_08_hurricaneIsaacCrisisMapScreenShot_600.jpg" style="width: 600px; height: 386px;" /></a>
<p class="caption">After Hurricane Isaac 2012, MapFasten was used to align this map of
the New Orleans levee system</p>
</div>
<p>We deployed MapFasten on Google App Engine in late 2012 but it will remain
in closed beta testing until we can devote some time to making it more
scalable and adding simple access controls so each user has their own
sandbox to work in.</p>
<p>Lots of interesting design decisions went into building the app and I
might go into some of them in future posts. In the meantime, if you're
interested in our take on the process of manually aligning maps, you can
check out how our interface works in the tutorial video.</p>