Etowah River Run

Today’s Etowah River Run was a hot and humid affair.  I chuckled as I read the weather report on my Droid near race time.  80 degrees and 87% humidity seemed right, but it was the NWS Heat Advisory that caught my attention: “stay in an air-conditioned room and stay out of direct sunshine.” Seemed as good a time as any to run a 5K!

We got a quick perspective check, though, as backpack-clad 24-hour racers from the nearby Gold Rush 24 Adventure came running by just before the start of the 5K.  We 5K’ers cheered them on heartily: cheers of encouragement, and of celebration that we would be done 50 times sooner.

Yet the course was flat and fast, the race is for a good cause, and it has become quite a tradition around Canton.  Turnout was good in spite of the heat, as this is race gets good participation from local cross-country teams (it falls in the training season, just before Saturday meets begin).  This was the second year that Lydia and I ran it, so we knew what to expect.

The one twist to this course is that the first mile is the fast one, with a long downhill.  This means it takes longer to “thin out” the crowds, and runners don’t fall into position until after mile 1.  A lot of folks let that pull them into too fast a pace and end up paying for it in the last half.  I reminded Lydia beforehand not to do that; she managed her pace well, and was strong at the finish.  And what an exciting finish!

After I was done, I stayed near the finish line and watched for Lydia.  As the timer cleared 27 minutes, I walked over and checked the card baskets.  There was nothing yet in the under 10 girls basket, so I knew if Lydia finished at her usual pace, she could grab a first place age group award.  I soon saw her running in, looking great, in first place!  About 20 yards behind her was a girl about her age gaining on her.  She overtook Lydia, and Lydia kicked into a faster sprint and passed her again.  It went back and forth until they passed the finish line side by side, shoulder to shoulder.  It was a dead tie as far as I can tell.  But, while in the chute, the girl grabbed a place card first, which put her in first and Lydia in second.  Lydia was a bit disappointed, but it was the most exciting finish in the race (yes, I’m a bit biased).

Officially, Lydia placed second at around 29:20.  I ran it in 24:35, and was happy enough to run a sub-8 5K while under a heat advisory.

After Lydia collected her award, we prepared to leave, but she then asked to stay for more door prizes.  Great suggestion!  I had forgotten how good and plentiful the prizes were at this race. Lydia won a $25 Walmart card, and I won a $50 Phidippides card.  And the race T-shirt is a good one.  It’s cotton, not technical, but it’s a nice design and an improvement over previous years.

Overall, it was a great morning, and we’ll run it again next year.  Surely, it will be cooler.

Painted into a Corner

I’ve had such good success with DB2 9.7 LUW under 64-bit Linux that I didn’t hesitate to install it on my new development laptop (Windows XP 32-bit, the corp-rat standard).  I immediately upgraded to Fix Pack 2, of course: never install fix level zero of anything.  I created and loaded a few databases and was on my way.

But it didn’t take long to notice some very bad behaviors.  I saw frequent CPU, memory, and I/O spikes, mostly in db2syscs and db2dasstm. On two occasions, all database operations froze without recovering, and db2syscs suddenly became a diehard: I couldn’t kill it from the Service Control Panel, Task Manager, or even pskill.  Windows shutdown then froze, requiring a hard power off.

This sent me on a fact-finding mission.  First stops: the Windows Event Viewer and db2diag.log.  Since I had not changed diagpath, I had to remind myself that instance information files are no longer under SQLLIB (with Windows, they’re now under Documents and Settings\All Users\Application Data\IBM\DB2). I spotted a huge and growing db2diag.log, at diaglevel 3.  It was flooded with messages like:

The db2diag flooding would certainly account for the CPU, I/O, and memory spikes, but I’m not sure about the periodic freezes.  But it’s one thing at a time, and couldn’t stop until I had a clean db2diag.

Fortunately, I found a fellow victim who reported the same issue just yesterday.  The root cause was that the DB2 Admin Server (db2dasstm) did not have required authorities on toolsdb.  This was surprising, since I let db2setup create and configure it.  But I’ve been accustomed to chanting the grant dbadm mantra since the introduction of SECADM, so I typed it out.

But in this case, it wouldn’t work. I couldn’t get into toolsdb with an authorization ID that had SECADM in order to do the grant. I tried stepping into it first via SYSADM/SYSADM_GROUP, but no dice. And toolsdb was unique in that SECADM was only granted to some bogus “SYSTEM” ID.  Thank you, db2setup, for painting me into a corner!

To fix it, I had to drop and re-create the toolsdb, following the proper steps to keep DAS happy.  A couple of db2admin stop/starts later and I had a small and steady diag log.

Time will tell if additional problems remain that contributed to the freeze problem (so far, so good), but I learned an important lesson: never let the DB2 installation program create the toolsdb.

Severe but Curable

On average, over 22,000 children die each day from preventable, hunger-related causes.  This fact that receives so little media attention should be among our most urgent priorities.

Over half of these deaths occur on the African continent, so as the World Cup turned attentions there, I sought answers to why the tragedy of hunger persists so cruelly and what can be done about it.  My reading included Richard Stearns’ The Hole in Our Gospel and Thurow and Kilman’s Enough: Why the World’s Poorest Starve in an Age of Plenty.  Taken together, these provided a well-rounded mix: the heart and perspective of a luxury goods CEO turned children’s champion, and pragmatic detail from on-the-ground Wall Street Journal correspondents.

Structural

Thurow and Kilman describe the structural problems that give rise to African hunger.  Bad policy decisions by governments, the World Bank, USAID, agricultural councils, and other groups lock many African nations into a cycle of dependency.  US crops that are dumped on the world market at heavily-subsidized prices undercut African farmers, driving them out of business.  Insistence on food-only aid exacerbates this problem; for example, during the 2003 Ethiopian famine, international aid trucks drove their loads of US grain past warehouses that were overflowing with the same unsold, unused staples grown by local farmers.  Approachable infrastructure problems lie festering.  More recently, subsidized Ethanol production diverted food to fuel, driving grain prices out of reach of the world’s poor.  As a result, the Green Revolution which decades ago lifted Asia out of hunger and poverty still evades Africa.

Spiritual

Richard Stearns decribes how it’s a spiritual problem.  He reminds us of our scriptural obligations (such as Matthew 25 and Isaiah 58) to provide for the poor and feed the hungry.  A gospel with a hole in it is one that accepts personal salvation but ignores the parables of the Good Samaritan, Lazarus, the Sheep and the Goats, and the 2,100 other Bible passages about caring for the poor.  Stearns describes how that conviction led him to leave a life of luxury and head World Vision.

Attainable

Spiritual and structural.  They’re both right.  And the solutions are well within reach.

Modifying food aid programs to allow local purchase may reduce US farm subsidies, but it will help African farmers become self-sufficient.  Providing African farmers the same financial security measures (such as futures and crop insurance) that farmers in developed nations enjoy will help shoulder their risks and smooth out the boom and bust cycles.  Relatively small investments in infrastructure (dams, irrigation, higher-yield seeds, and road improvements) can dramatically improve farming productivity.  Simple treatments and immunizations can very effectively prevent and confront the crises of malaria, tuberculosis, and AIDS.  Small yet powerful innovations like Plumpy’nut and the KickStart pump can be rapidly deployed.   These proactive and preventative measures are not only affordable, they’re collectively less expensive than remedial aid has been.

Making a Difference

But most of us can’t hop on a plane tomorrow to go build a dam or start a futures exchange in Africa.  So what can we do?  As Mother Theresa has said, “we cannot do great things on this Earth, only small things with great love.”

For some quick encouragement and ideas, read Chapter 10 of Enough.  If you don’t already support World Vision or a similar ministry, sponsor one or more children from their web site.  I’ve seen first-hand the effectiveness of World Vision’s work in the developing world, and I highly recommend them.  And if budgeting is an issue, consider your very prominent position on the Global Rich List; for example, if you make $35,000 annually, you’re richer than 95% of the world.

Global hunger is a large-scale, often overwhelming problem, but that’s no excuse for paralysis or inaction.  It can be solved, one step at a time, one person at a time.

Woodstock Freedom Run

The annual Woodstock Freedom Run is perhaps my favorite 5K, mainly because it benefits The Hope Center.  It certainly helps to have Woodstock charm, a flat course, and gracious weather for July (around 70 degrees at race time).  And it’s always a nice surprise to chat with so many folks that I don’t see often.  It feels like a family reunion.

Lydia won a second-place age group award at 28:09.  Stephen ran very well at 30:06 and was all smiles as he crossed the line, but he’s in the 11-14 group now, competing with 18-minute finishers.  At 25:32, I was certainly no threat to 15-minute defending champ Sammy Nyamongo, but enjoyed the race all the same.

It should return to July 4 next year, so if you’re not running the Peachtree, come out for this one.

Contrast

I happened to notice that an SUnit I added today was number 1,000.  Perhaps I should get free groceries for that milestone, but it reminded me just how much and why we take xUnits and test-driven-development (TDD) for granted.  I was a bit more aware, having spent yesterday working on a customer’s C++ code which had no xUnits or test scaffolding.

It really was a study in contrasts.

For the customer’s C++ code, I found myself fighting in the Visual Studio debugger mostly with overzealous constructors and destructors (folks, just instantiate and delete objects in constructors/destructors, don’t use them for wild stuff like reading registries, calling decryption libraries, and connecting and disconnecting from databases).  But the real hassle was having to run the code-compile-link-bounce-deploy gauntlet far too many times.  Often this isn’t bad (yes, incremental compile and auto build all help), but in this case, it took a lot set-up time getting data in the state it needed to be in before calling this code, and every code change required repeating that.  That’s usually true even of hot swap JVMs.

Compare that to my SUnit-driven VA Smalltalk code.  I wrote the test case and my first stub method before I wrote the code.  I ran the test, and of course, it failed (failed big, too: red, not yellow, due to an expected message not understood exception).  I re-ran the SUnit with Debug, and started filling in the code – in the debugger.  I added methods and even an entire class that didn’t exist before I started debugging.  I inspected objects and ivars and even changed a few to guide me through crafting and whittling at the code.  I ran it again and got to yellow.  One more “code it while debugging it” pass and the test went green.

Red, then yellow, then green.  My favorite development process.

I’ll soon upgrade my Visual Studio 2008 to 2010 and look forward to the new features it offers (thank you, Microsoft for finally bundling ASP .NET MVC, and for abandoning your own JavaScript kluges and just adopting jQuery).  But, decades later, it’s still nowhere near as productive as VA Smalltalk, VisualWorks, and Pharo, where you can write the code in the debugger while the system is still running and never have to stop it.

Why haven’t we learned?

Barracuda Attack

Like the song’s antagonist, one of my two Barracuda 7200.12 1 TB hard drives just became a real low-lying nuisance.  Yesterday, my four-way Windows 7 box grew incredibly sluggish, a problem which persisted over restarts.  With the case cover off, I could hear intermittent clicks and buzzes coming from the primary hard drive.  They weren’t the extreme clicks of death, but sounds that seemed to indicate stumbling over chunks of bad sectors.

This was one of those once-every-year-or-so moments when backups go from pure overhead to the greatest thing since the ten cent gigabyte.

I suppose there are as many backup strategies as there are computers.  Components of this strategy balance the trade-offs between redundancy and recovery time, with RAID-1 or RAID-10 at one end, and incremental backups of just the personal data at the other.  I’ve gotten out of the RAID-1 habit for a few reasons: consistency with laptops, bad experiences with “logical” data failures (if you corrupt a Windows registry hive, you’ve just backed up the problem), and problems with lower-end RAID controllers.  But for my quick recovery needs, I do perhaps the next closest thing: put in pairs of drives and use periodic imaging and regular XXCOPY script runs (with /CLONE) to keep them in sync.

Because of this, recovery was indeed quick and easy; I simply swapped my shadow hard drive to primary and restarted.  This took the bad drive out of the critical path so I could find out what was wrong, and perhaps RMA this thing that I’ve only had for six months.

A full chkdsk found no bad sectors, and the SMART stats were within thresholds (with SMART, the absence of bad news is not necessarily good news).  Confounded, I let HD Tune run a full error scan.  It reported no bad sectors, but I could again hear the clicks and buzzes at various points: around 250 GB, 300 GB, 568 GB, 596 GB, and 695 GB.

So maybe this drive is farther from full failure than I thought.  But I’m not taking my chances, especially given Seagate’s checkered past with drives in this line.

Ironically, on the same day of the failure, I received a TigerDirect email offering this exact drive for $69.  Pass.

I Ignore Your Puny 32 Bits

Customers these days: they think they’re entitled to 64 bits.  Heck, I remember when we had to do everything with just 32 bits, and only half of those worked.

Since 64-bit Windows and Linux are the new normal, the DB2 LUW folks (particularly with 9.7) have been encouraging migration away from 32 bit apps and libraries.  Sometimes this is more than a gentle nudging, as things break and require intervention.

Colleague Allen Cartright shares a tip for managing 32-bit ODBC data sources when 64-bit Windows just wants to ignore them.  So, if you find yourself running 32-bit DB2 code on a 64-bit OS written by a 2-bit company that can’t stand 1-bit of competition, heed his advice:

The ODBC connection manager in Windows Server 2008 (Administrative Tools -> Data Sources) is 64 bit only.  To get to the 32 bit ODBC connection manager in Windows Server 2008 Rx x64 you must run the executable directly as there aren’t any icons for it.  The executable is “C:\Windows\SysWOW64\odbcad32.exe”.  This is the same executable name as the 64 bit version but is in the legacy directory.  Once you have the 32 bit version of the ODBC Connection Manager open you may add the ODBC connections as normal.

The Microsoft Knowledge Base post on this has additional information and recommends a useful naming convention:

To work around this problem, use the appropriate version of the ODBC Administrator tool. If you build and then run an application as a 32-bit application on a 64-bit operating system, you must create the ODBC data source by using the ODBC Administrator tool in %windir%\SysWOW64\odbcad32.exe. To indicate the type of DSN, you can add “_32″ to the 32-bit user DSNs and “_64″ to the 64-bit user DSNs.

Between the Lines

Today I was asked for some quick help with additions to a customer’s C++ code running as a user exit in one of our systems.  This bit of code used sprintf to format an error message into a buffer passed in by our system.  Pretty pedestrian stuff, but it came among many surprises, one being that the code often had no regard for buffer lengths, and risked overflowing them.

The buffer was only 256 bytes because it’s used for a summary message (if there’s more to say, it should be said elsewhere, such as a log file).  256 bytes is pretty easy to overflow, particularly with additions we added, so we needed some crash protection.  snprintf would have been a quick alternative, but was not available on this platform.  So I added widths to the format specifiers, changing something like this:

sprintf(pszMessage, “SQL error in whatever, rc=%04d, SQL=%s”, rc, pSql->Stmt);

to this:

sprintf(pszMessage, “SQL error in whatever, rc=%04d, SQL=%.215s”, rc, pSql->Stmt);

To avoid magic numbers, I could have done something like:

sprintf(pszMessage, “SQL error in whatever, rc=%04d, SQL=%.*s”, rc, (BUFFER_SIZE – 40), pSql->Stmt);

And had snprintf been available, it would have been even cleaner:

snprintf(pszMessage, BUFFER_SIZE, “SQL error in whatever, rc=%04d, SQL=%s”, rc, pSql->Stmt);

Truncating a message is not ideal, but it sure beats tossing a GPF.  In this case, losing the end of the SQL statement would not be a big deal, but the message could be rearranged or additional logging added if it were.

The point is that too much C and C++ code in the wild holds too little concern for bounds checking.  Fortunately, I now code mainly in OO languages where this is not a concern – things grow as needed.  But since C and C++ (taken together) are still arguably by far the most pervasive programming languages (with new programmers on board with it every day), it’s very much relevant.  The advice is simple:

Avoid unbounded string functions.  Avoid sprintf, strcpy, strcat, strlen, and gets.  Use snprintf, strlcpy, strlcat, strncpy, strncat, fgets, and similar functions depending on availability.  If these alternatives aren’t available, use techniques like the above to avoid overflowing buffers.  When using the strn functions, remember to leave room at the end to add the null terminator (and remember to add it!)

You can find further such advice in the C Programmer’s Ten Commandments.

Back in the day, when I coded mostly in C and C++, we augmented our lint process to scan for things like strcpy, strcat, and sprintf.  Richer compiler warnings have largely taken the place of lint (with extensions lost along the way), but it’s easy enough to simply grep for these.  And the time spent staying between the lines is a small price to pay for some crash insurance.

The Friday Fragment

It’s time again for the Friday Fragment: my weekly programming-related puzzle.

This Week’s Fragment?

I caught a little grief for inflicting Lisp on readers last week.  So I took that (along with sunny evenings, good river water levels, race targets, and a poolside reading list) as a sign to give the Friday Fragment a summer break.  I have some ideas for new types of challenges when we pick it up again, so look for its return in the fall.  Until then, enjoy the summer!

Last Week’s Fragment – Solution

Last week’s fragment was a challenge to think inductively and recursively:

Code a factorial function (s-expression) in Lisp or Scheme.  For example, (factorial 3) should evaluate to 6.  Include unit test scaffolding code.

As mentioned above, the response to coding in Lisp/Scheme wasn’t positive.  One comment I got was “I’d rather write Esperanto poetry”  (kids these days, can’t think outside the box).  So I was left to code this one on my own.  Here’s my implementation:

   (define factorial
         (lambda (n)
             (cond
                 ((< n 0) "n must be non-negative")
                 ((= n 0) 1)
                 (else (* n (factorial (- n 1)))))))

My test/scaffolding code is:

   ; Unit test case helper:
   (define test
     (lambda (x y)
       (cond
         ((equal? x y) "pass")
         (else (list "FAIL - expected: " y " but got: " x)))))

   "Factorial unit tests"
   (test (factorial -1) "n must be non-negative")
   (test (factorial 0) 1)
   (test (factorial 1) 1)
   (test (factorial 3) 6)
   (test (factorial 55) 12696403353658275925965100847566516959580321051449436762275840000000000000)

I used Dr. Scheme (PLT Scheme) for this, which (like most functional programming environments) is optimized for tail recursion.  So even large factorials run fast without error.  By comparison, recursive factorial functions in C and Java will (depending on argument size, stack size, and/or VM) either take forever or fail with a stack overflow.  One solution for these more traditional environments is to convert tail recursion to iteration and avoid the "stack attack" of recursive function calls.

Battling Droids

Given recent questions about my Droid phone, perhaps it’s time to post again about it.  This time, I’ll offer tips on battling two common Droid demons.

1. The condensation poltergeist. The tiniest amounts of condensation can make Droid’s touch screen act possessed.  The Ghost of Droid will scroll automatically, start apps, search for things, make phone calls, and wreck all sorts of direct-manipulation havoc.  And there’s no point in fighting it: it’s much faster than you, and when it takes over, it’s usually impossible to win the battle and override it.  A locked screen offers some protection against emailing your boss or calling Tokyo without your consent; in this case, it can only repeatedly try to draw out your unlock pattern, usually resulting in a series of “wait 30 seconds” holds.  It’s entertaining to watch, but annoying to say the least.

This weird phenom has been ascribed to viruses, chargers, and other hardware and software issues, but, in my experience, it’s always due to condensation on the touch screen.  And since it requires so little moisture, it’s hard to predict when it will happen.  It has happened to me upon walking into an indoor pool area, and inside my car after a long run.

As long as you keep your Droid at 70 degrees and 40% humidity (perhaps in the raised floor area of your personal data center), you’ll be fine.  For those of us in the real world (and who like Georgia summers), just stop using it when it happens, turn it off, be patient, and wait for it to dry out.  You can help it along by bringing it back into the air conditioned indoors or using a blow dryer on a cool setting.

2. The grim reaper ringer. That “slide to answer” control and dexterity test works great at preventing accidental unlocks and butt calls.  So great that if you try it while driving, you’re as likely to annihilate as answer.  An early Android update made it slightly easier (straight rather than curved slide), but it’s still difficult.  Most bluetooth headsets provide an alternative, but if you’re not headsetting it, it’s best to just pull over before attempting to answer, or simply miss the call and return it later.  Answering that call is not worth crashing into the ditch or oncoming traffic.

I’ve heard that the latter problem will be fixed soon in an Android (software) update, but the touch screen issues will probably have to wait for a new phone (gotta love “new every two”).  My next phone will almost certainly be an Android, but probably won’t be a Motorola Droid.

The Friday Fragment

It’s Friday, and time again for a new Fragment: my weekly programming-related puzzle.

This Week’s Fragment

With this week’s fragment, we’ll start a series of puzzles to code something in a specified language.  The code itself won’t be difficult, but it will highlight the unique benefits of a particular language (or, in some cases, recently-added features to a language).  It’s your chance to try out a new language or feature.

Will start by thinking inductively and recursively, and that’s where functional languages like Lisp/Scheme shine.  This is a “classic” exercise (meaning there are probably a million solutions posted on the web), but try it yourself before peeking.  The problem is simple:

Code a factorial function (s-expression) in Lisp or Scheme.  For example, (factorial 3) should evaluate to 6.  Include unit test scaffolding code.

For extra credit, code a recursive factorial function in a more mainstream language (like C or Java) and compare the performance/results to Lisp when taking the factorial of a very large number.  There are several free Lisp/Scheme implementations you can use; Dr. Scheme (PLT Scheme) is one good choice.

To “play along”, post the solution as a comment or send it via email.   To avoid “spoilers”, simply don’t expand comments for this post.

Last Week’s Fragment – Solution

Last week’s fragment was sent in by Joel Odom:

Write C code or pseudo-code to swap the values of two integer (int) variables, a and b, without declaring or using any other variables. Do not use any syntactical language tricks that implicitly copy values out of a or b into other memory or registers.  You can use C operators, control flow, and similar statements.

I was left to solve this one on my own, and sent Joel two solutions.  For the first, I used the wonders of XOR.  Among other things, XOR’ing something with a value twice gets the original value back: a feature used for simple bitmap animation, masking, encryption, etc. – and it’s fast.  For the second, I called a function, using the stack to push/pop one of the values.  Joel approved the XOR approach, but said the function call was cheating since it allocated memory on the stack (good point).  Here’s the simple XOR solution; for the function call and scaffolding code, see comments to last week’s post.

a=a^b;
b=b^a;
a=a^b;

Defaulting Less Security

Today, a friend reported that one of the apps I provide as a community service was down.  Its WebCalendar component complained that magic_quotes_gpc was no longer enabled, which I quickly confirmed by dropping in a phpinfo() call.  The remedy was also quick and easy: add a local php.ini with:

magic_quotes_gpc = On

This automatically adds slashes to escape quotes and other characters in GET, POST, and Cookie strings (hence “gpc”).  The PHP code then removes these via stripslashes and similar techniques.

Magic_quotes_gpc is no longer considered a good way to guard against SQL injection attacks, so the PHP Security Consortium and others now recommend against it.  I suspect this is why my hosting service changed their global setting to off, but security-wise, that was a step in the wrong direction.

Many PHP apps that support magic quotes are coded to work even if it’s turned off, and the get_magic_quotes_gpc()  ? stripslashes template for doing this is seemingly everywhere.  Fortunately, WebCalendar checks for this, but many apps don’t.  Disabling magic quotes was probably done to force apps to change, but it’s more likely apps will continue to work and developers and admins won’t realize that they are suddenly far more susceptible to injection attacks.  A better approach would have been to just let it die with the 5.3 upgrade.

The Friday Fragment

It’s Friday, and time again for a new Fragment: my weekly programming-related puzzle.

This Week’s Fragment

This week’s programming puzzle was sent in by Joel Odom:

Write C code or pseudo-code to swap the values of two integer (int) variables, a and b, without declaring or using any other variables. Do not use any syntactical language tricks that implicitly copy values out of a or b into other memory or registers.  You can use C operators, control flow, and similar statements.

If you want to “play along”, post the solution as a comment or send it via email.   To avoid “spoilers”, simply don’t expand comments for this post.

Last Week’s Fragment – Solution

Last week’s puzzle was the following programming-related cryptogram:

Baae jaanklb xpnrv xksr.  Ki gaq pwr sper xa cpkx, kx kv xa vrwhr gaq urxxrw, ple xa ftrpvr gaq.

Congratulations to Luke for solving this; the plaintext answer is:

Good cooking takes time.  If you are made to wait, it is to serve you better, and to please you.

This is a chapter intro quote in Fred Brooks’ classic The Mythical Man Month, taken from a French restaurant menu.  It reminds us that quality software also requires time and careful preparation.

Go For It

I’ve been exploring new concurrent programming approaches lately, and decided to give Google’s new Go programming language a try.  It’s fast, straightforward, and a good fit wherever you might use C or C++,  such as systems programming.  It borrows a lot from C (including syntax, pointers, and the gcc compiler), but adds significant improvements, including garbage collection and its own flavor of coroutines called goroutines.  “Goroutines” is certainly clever, but I’ll never understand why Google gave its creation the decidedly ungoogleable name of “go” (hence the alternate name, golang: so you can find it).

To get started, I had to check out sources from mercurial and build them, but the instructions were clear and complete (“cookbook”, as we say).  They were written for Linux (complete with sudo apt-get installs for gcc, bison, and other pre-reqs), so I used Ubuntu.  I suppose it’s possible to develop under Windows with Cygwin, but I didn’t try it.

The emphasis on clean, lightweight, and terse code is clear throughout.  For example, it takes a simple approach to common OO mainstays: it uses structs, types, and interfaces instead of classes and inheritance (like C#, structs can have methods); new and make allocators with automatic reference counting; and a built-in map syntax instead of dictionary or hash table classes.  It provides syntactic sugar for common shorthands, such as := to declare and initialize a variable in one step, permission to omit semicolons and parentheses, and code-reducing conveniences like returning multiple values from a function.  It uses Java-style packages and imports for organizing code and supports Python-inspired slices for array references.

Goroutines are surprisingly easy to use.  Any function call can be invoked as a goroutine (forked) simply by prefixing the call with the keyword go.  The function itself can be inner, like a block or anonymous function.  Channels provide the communication and rendezvous vehicle (like a future and queue) via a unique and simple syntax.

There’s a lot missing from the language (much of that deliberate, like classes and inheritance), and it lacks maturity and production readiness.  But it’s easy to appreciate how its primary inventions (goroutines, channels, maps, slices, return lists, and implied interfaces) work together to form a clean, cohesive toolbox for fast concurrent programming.  While I don’t foresee using go for “real work”, I hope to see it influence other environments which often make parallel programming harder than it has to be.

The Friday Fragment

It’s Friday, and time again for a new Fragment: my weekly programming-related puzzle.

This Week’s Fragment

This one’s for Luke, who enjoyed our last cryptogram.  Solve the following programming-related cipher:

Baae jaanklb xpnrv xksr.  Ki gaq pwr sper xa cpkx, kx kv xa vrwhr gaq urxxrw, ple xa ftrpvr gaq.

This uses simple substitution, so just plug at it with pencil and paper, building up the substitution table as you go.  For extra credit, tell us the source of this quote.  If you’re ambitious, try coding a cracker, or at least a cryptogram generator.

If you want to “play along”, post the solution as a comment or send it via email.   To avoid “spoilers”, simply don’t expand comments for this post.

Last Week’s Fragment – Solution

Last week’s challenge was to write code to solve a recent Car Talk Puzzler; that is:

Write code to find two (two-digit) temperatures where the (rounded) Celsius and Fahrenheit readings have the same digits, only reversed.

The conversion to Fahrenheit is 1.8 (or 9/5) times Celsius plus 32.  If you round the result, you find that 16 Celsius is 61 Fahrenheit and 28 Celsius is 82 Fahrenheit: handy numbers to remember when you want a quick approximate conversion from the current temp.

I invited folks to code it in multiple languages to compare, and even try esoteric languages.  Joel coded it in Python and demonstrated the strength of its range function when used with iterators.  He also showed off reversing a string via a “slice with inverse stride”, which sounds like my golf technique.  Wayne’s Smalltalk example demonstrated how it can elegantly iterate over a range using blocks/closures, a technique and style so nice that Ruby ripped it off.  Spencer coded it in Shakespeare but, alas, had problems with the interpreter (it was not to be).  I tried LOLCODE , and likewise had interpreter problems, so I did a quick PHP implementation also.  Gee, I wonder why there aren’t any production-ready LOLCODE or Shakespeare compilers?

Thanks to all for posting; code fragments like these provide good opportunities to compare language features and learn new approaches.  For all these implementations, see the comments in last week’s post.

CLPMinus

There are many great tools for running DB2 commands; I often use TOAD, Control Center (db2ce), and a CLI-based tool I wrote.  And with DB2 9.7, I’ve enjoyed experimenting with the new CLPPlus, IBM’s answer to SQL*Plus for Oracle refugees.  But for those quick commands, I usually just pop open a DB2 Command Window (Windows) or bash shell (Linux) and type “db2″ followed by the command.  It works great nearly all the time.  Emphasis on nearly.

Today, Wayne reported how this command taken verbatim from the DB2 documentation (the “File type modifiers for export” section) choked:

db2 export to delfile2 of del modified by timestampformat=”yyyy.mm.dd hh:mm tt” select * from schedule

It failed with the error: SQL3192N  In the filetmod a user specified format “TIMESTAMPFORMAT” beginning with the string “yyyy” is not valid.  And no value for timestampformat worked.

This is a case where bugs in CLP’s command-line parsing (particularly with double quotes) get in the way.  The command works from inside CLP, Command Editor, and other tools, so you can just type db2 or db2ce and hit enter and then run the command (without “db2″ in front) from there.  Using admin_cmd from a CLI/ODBC tool also works, like so:

call admin_cmd(‘ export to delfile2 of del modified by timestampformat=”yyyy.mm.dd hh:mm tt” select * from schedule’)

Bugs like this have survived in the db2 command line for awhile (this fails even in 9.7).  I’ll report it, but since CLPPlus is the new kid in town, the old db2 command line probably won’t get as much attention.

The Friday Fragment

It’s Friday, and time again for a new Fragment: my weekly programming-related puzzle.

This Week’s Fragment

A recent Car Talk Puzzler challenged listeners to find two temperatures where the Celsius and Fahrenheit readings had the same two digits, but reversed.  For example, 28 Celsius is 82 Fahrenheit.  It’s an easy problem to solve by hand, but what fun is that?  This is one where writing a bit of code will yield an answer faster than working it out by hand.  Or, better still, offer a chance to try some new coding approaches.  So, our programming version of this puzzle is:

Write code to find two (two-digit) temperatures where the (rounded) Celsius and Fahrenheit readings have the same digits, only reversed.

Use any language you’d like; you may want to try it in multiple languages to compare.  You might even try an esoteric language like LOLCODE or Shakespeare.

If you want to “play along”, post the solution as a comment or send it via email.   To avoid “spoilers”, simply don’t expand comments for this post.  That is, unless you want to see what languages others used to deliberately try something different.  Language wars can be such good sport.

Last Week’s Fragment – Solution

Last week’s fragment was one of those puzzles sometimes given as interview questions for programming and engineering jobs:

You have eight balls; seven weigh the same, but one is lighter than the others.  Using a balance, how can you find the “oddball” in only two weighings?

Congratulations to Wayne who solved it swimmingly, and described the solution nicely:

Weigh three balls against another three balls. If they weigh the same, you simply weigh the other two to find the lighter one. If one set of three balls was lighter than the other though, weigh two of the balls in the lighter set of three against each other. If one is lighter, that’s it, else it’s the third one you didn’t weigh the second time.

I guess that means you’re hired!

DML Insurance

In a chat today, we discussed how to protect data when formulating DML, particularly those SQL update and delete statements that can destroy perfectly good data if done incorrectly.  You may find yourself having to run such SQLs against large data sets (and, yes, even production data), so no amount of care is too much.

One option is to hold a transaction open, run the DML, select the (uncommitted) results, and then commit if good or rollback if not.  That’s clever, but it can hold a lot of locks for a long time, is error-prone, and doesn’t work for all cases.  Another is to keep full backups handy and restore the table(s) if something goes wrong.  This can be time consuming and can harm availability during restore if the table(s) are large.

A better approach is to create exports of exactly the data you will be changing or deleting.  That is, run an export command whose where clause is exactly the same as that in your update or delete statement.  That way, you can restore (via import) just the data you changed.  For DB2, I use IXF exports since they’re the richest.  So, for example, before you:

update company.employees set salary=100000000 where name=’Derek’

You should:

export to empderek.ixf of ixf select * from company.employees where name=’Derek’

If you realized maybe you shouldn’t have done that, you can put it back with:

import from empderek.ixf of ixf insert_update into company.employees

This can be used to recover from deletes as well (hence insert_update).  For example, before you:

delete from wikipedia.articles where title=’Malamanteau’

You should:

export to articles.ixf of ixf select * from from wikipedia.articles where title=’Malamanteau’

MySQL has fewer export/import options, but mysqldump can definitely help.  For example, export with:

mysqldump wikipedia articles –compact –where=”title=’Malamanteau’” > articles.sql

And, after the above delete, you can put it back with:

mysql wikipedia < articles.sql

For ease of reference, I often give these “backup” IXF and SQL files unique, descriptive names. It’s helpful to include the date and some tag that identifies what the data is.  In the above example, I could have used articles-Malamanteau-12-May-2010.ixf, although Malamanteau is hard to spell and probably isn’t even a real word.

Perhaps this simple technique of “export first, DML second” may rescue you from the occasional “oops moment.”

Sorting it Out

In prior posts, I described some of the benefits of DB2’s Self-Tuning Memory Manager (STMM), along with a few caveats.  One caution was around automated sort heaps, since sort requirements can often be unpredictable and confusing.

High-volume OLTP systems can, at times, experience heavy loads of small sorts or sort-like activity (hash joins, grpby’s, etc.).  It’s not uncommon for many of these to overflow (spill), or for a sub-optimal access plan to arise, even when more than enough memory is available for sorts.  Like the proverbial bad check writer, you’re left thinking, “I can’t be overflowing, I still have memory left!”

Sort overflow symptoms include high I/O and CPU utilization, poor response times, high sort overflow snapshot counts, long sort times, and large numbers of rows written on read-only SQLs like SELECTs.  Sort heap tuning is often one of the “last frontiers” in performance work (after fixing low-hanging fruit like SQLs, indexes, and bufferpools), and can be a stubborn one.

From my experience, most sort overflow problems in DB2 9.x fall into these categories:

1. SQL.  In high-volume OLTP systems, it’s important to avoid sorting as much as possible.  Developers usually recognize that order by clauses require sorts, but can miss more subtle uses of the sort heap.  Count, distinct, and group by clauses also often require sort heap space, as do hash joins.  Statement snapshots and explains will reveal these.  The first step should be to try to rewrite problem SQLs to eliminate the sorts.  But wherever that isn’t possible, try step 2:
2. DDL.  Adding or changing indexes and clustering may eliminate sorts.  If an existing index is used but sorting still occurs, check to see if adding index or include columns will help, and verify that allow reverse scans is specified when needed.  Again, the only way to know for sure is to measure by snapshots and explains.  In some cases, MDCs can help, but check it first by “doing the math” or running db2advis.  Sometimes the problem is as simple as outdated statistics (among other things, outdated statistics can cause DB2 to request too small a sort heap), so make sure your runstats is occurring as frequently as needed.
3. Configuration.  And here we come to the STMM benefits and caveats.  You should “go automatic” with sort heap sizes, but be prepared to monitor and adjust as needed.

STMM only tunes shared sort memory, so to enable self-tuning sort memory, you must:

• Set the instance (DBM)-level sheapthres to zero (0).  You can verify this with: db2 get dbm cfg  | grep -i sheap
• Set the database-level sheapthres_shr (total amount of sort memory) and sortheap (memory available for each sort) to automatic.  You can verify this with: db2 get db cfg for <database> | grep -i sort

In some cases (such as when you’re experimenting and don’t want to wait on STMM’s tuning cycles), you may want to set an initial value for sortheap.  To do this, specify it alongside the automatic parameter, like so: db2 update db cfg using sortheap <size> automatic immediate.   But the initial value matters little after STMM “warms up” and then saves its settings.

When monitoring a database with sort overflow problems, keep an eye on the percentage of overflows (under 3% is a good target) and on the current sortheap and sheapthres_shr sizes.  You can view these with the snapshot monitors (db2 get snapshot for all on <database> | grep -i sort) and database configuration (db2 get db cfg for <database> show detail | grep -i sort).   But you may find, as I often have, that frequent, small overflows are occurring even when the sort memory areas have stretched to more than enough space.

Why this happens is a mystery to me.  A parallel symptom I often see is that the sort heap size reported in explain (db2exfmt) outputs is often far smaller than the current sortheap size.  At this point, the only choices are to leave sort heaps at automatic and accept some overflows, or abandon automatic and set a fixed size.  When setting large fixed sizes, remember that an overflowed sort writes the entire sort heap to tempspace.  So, a very large sort heap may coax the optimizer into relying more on sorts, only to have some of these very large sorts spill everything.

IBM is considering allowing ceiling and/or floor values for certain automatic parameters.  I do hope this gets added in a future DB2, and that sort heaps are among the first that can be configured this way.  Until then, DBAs will be left to sort this out on their own.

The Friday Fragment

It’s Friday, and time again for a new Fragment: my weekly programming-related puzzle.

This Week’s Fragment

This week’s fragment follows the same line as last week; that is, it’s a simple puzzle sometimes given as an interview question for programming and engineering jobs.  I once worked at a company that used this one.  Such puzzles are usually very poor predictors of job performance, but they’re interesting nonetheless.  So here goes…

You have eight balls; seven weigh the same, but one is lighter than the others.  Using a balance, how can you find the “oddball” in only two weighings?

If you want to “play along”, post the solution as a comment or send it via email.   To avoid “spoilers”, simply don’t expand comments for this post.  It should be a ball.

Last Week’s Fragment – Solution

Last week’s fragment was borrowed from a recent Car Talk episode.  A listener there wrote how he didn’t get a programming job partly because he failed to solve the following interview puzzle within the allotted five minutes:

You have a four ounce glass, a nine ounce glass and an unlimited supply of water, which can be poured and dumped as much as needed.  Can you measure exactly six ounces of water in as few steps as possible?

Kudos to both Spencer and Luke for solving this one quickly (within the five minutes) by just talking through it.  It can be done in eight steps, and the key is recognizing that the six ounces clearly must end up in the nine-ounce glass.  So you have to work toward pouring exactly three ounces out of that glass.  Get one ounce in the four-ounce glass and you have your three-ounce destination.  Here are the steps.

1. Fill the nine-ounce glass.
2. Fill the four-ounce glass from the nine-ounce glass.  That leaves five ounces in the niner.
3. Dump the four-ounce glass.
4. Fill the four-ounce glass again from the nine-ounce glass.  That leaves one ounce in the nine-ounce glass.
5. Again dump the four-ounce glass.
6. Pour that one ounce from the nine-ounce glass into the four-ounce glass.
7. Fill the nine-ounce glass again.
8. Fill the four-ounce glass from the nine-ounce glass.   You will have poured off three ounces leaving six ounces.  Voila!