Monthly Archives: January 2010

Droid Does

A friend and fellow Droidian asked me today what my favorite free Android apps were.  I really didn’t know where to begin because I use so many of them so regularly.  This includes, of course, the preloaded ones: music, calendar, gmail, browser, maps, navigation, talk, contacts, messaging, YouTube, Picasa, etc., etc..  But to answer the question, I thought I’d jot down a list of some of my favorite free Market downloads here:

– Scoreboard – All my sports scores, all the time.
– Pandora and Slacker – Personalized internet radio, love it.
– Google Places – Must… find… coffee.  Perfect GPS compliment to Maps and Navigation.
– Flixster Movies – What’s on and what will be on.
– Weather Channel – Faster than looking out the window.
– Google Sky Map, Clear Sky Droid – For my astronomy fixes.
– Google Goggles – Still under development, but fun to try.
– androidVNC and ConnectBot (ssh) – The remote access classics.
– Shop Savvy – Barcodes R us.
– NewsRob, wpToGo, Google Listen, Read Later – For feeds, blogs, podcasts, and news.
– KeePassDroid – The world’s most portable password safe.
– Evernote – Snap a photo or jot a note, and it’s on the web in seconds.
– “Commercial news” readers: Thomson Reuters, BBC News, USA Today, Fox News, etc.
– Toys and Games – Compass, Labyrinth, DailyStrip, Jewels, Sudoku, Solitaire, Frozen Bubble, etc., etc.
– And, of course, lots of mainly pointless apps like Run, Zombie, Run and Talking Parrot.

Of course, half the fun is searching Android Market for any app you can think of that just might be out there.  There’s a good chance it is.

Ceol na Gael

Perhaps it’s that small bit of Irish blood in me, or simply that all my memories of live bagpipes are very pleasant ones, but good Irish music always brings good cheer.  Irish pipe band Ceol na Gael (“music of the Gaels”) from Charlotte, NC is especially skilled at it, and they continue to grow in fame and repertoire.

One of the band members, Wayne, is a friend and co-worker: a talented programmer and piper.  Check them out at their web site or Facebook page.  And if you’re a Carolinian in need of a good Irish fix, contact them.

STeMMing The Tide

The sheer number of DB2’s memory-related parameters can hit like a tidal wave.  Bufferpools, sort heaps, lock lists, package/statement caches, and various other heaps can all harm performance if undersized, and can waste or overcommit memory if oversized.  Further, many of the configuration parameters are interrelated; for example, increase logbufsz and you may also have to increase dbheap to accomodate it.  DBAs and performance tuners can feel overwhelmed by the number of knobs one must turn in each database.  And developers can feel overburdened when providing scripts and configuration tools to right-size their databases for each customer’s unique volume requirements.

Fortunately the latest (9.x) versions of DB2 LUW provide the Self-Tuning Memory Manager (STMM).  This feature significantly simplifies memory management in DB2, letting the database automatically size memory areas based on activity and needs.  For most parameters, you can also specify an optional initial size to avoid the overhead of progressively stretching memory after startup.

To turn on STMM for a database, specify:

update db cfg using self_tuning_mem on

You’ll likely also want database_memory set to automatic:

update db cfg using database_memory automatic

You can fix-size or initial-size database_memory, or use the computed option for back-compatibility.  The publib DB2 documentation has more information.

At this point, you can auto-size many of the memory areas.  For example, for dynamic lock list sizing, specify:

update db cfg using maxlocks automatic
update db cfg using locklist automatic

Depending on DB2 version, you can configure any of the following this way: dbheap, pckcachesz, sortheap, stmtheap, applheapsz, appl_memory, stat_heap_sz, dft_prefetch_sz, num_iocleaners, num_ioservers, maxappls, and avg_appls.   Anyone who has tried to package or build scripts that set these beyond the old (painfully too low) defaults can appreciate the benefits of “going automatic.”

Perhaps the greatest benefit comes when auto-sizing bufferpools.  For that, specify:

alter bufferpool <poolname> size automatic

To specify an initial (startup) size (in pages), place it before the automatic keyword.

In my experience so far, “going automatic” has worked swimmingly, with one exception: sort heaps.  OLTP systems can, at times, experience heavy loads of small sorts or sort-like activity (hash joins, grpby’s, etc.).  It’s not uncommon for too many of these to overflow (spill), or for a sub-optimal access plan to arise, even when more than enough memory is available for sorts.  This is partly because auto-sizing the sort heap requires some extra work and partly because the optimizer can’t predict future sizes.  More on this in a future post.

GWeaT New Release

I no longer actively use Google Web Toolkit (GWT), but some affinity remains.  I don’t miss experiencing its growing pains from 1.3 to 1.6: maintaining parallel hierarchies of cross-compilable business and transfer objects, forging and tweaking copy and mutation frameworks, debugging generated JavaScript, deployment issues, waiting seemingly forever for Java 5 and 6 support, and so on.  But spending trench time in any new technology has a way of fostering interest in seeing it mature.

And mature it has!

After a long stream of GWT updates with relatively small incremental enhancements, GWT 2.0 is a major leap forward, and a game changer.  It’s the GWT I’ve always wanted, and one I wish I had at the beginning.

Google has finally taken on the tools side of the equation with their own Eclipse plug-in.  Its operation is smooth and seamless: no more cobbled-up mash of third-party plugins, Cypal studio, and gwtCompile Ant tasks.

I’m thrilled to see the death of hosted mode; Google now offers native browser plug-ins for debugging.  I used the plug-ins for Firefox (3.5.7) and Chrome (4.0.302.3, dev channel) and debugging was a breeze, albeit sluggish at times.  Among other things, this ends the problem of code working one way in hosted mode and another way (or not at all) when deployed to a browser.

The Speed Tracer looks promising for performance monitoring and tuning.  However, I could not get past its “no data received in 6 seconds” error.  I used the latest dev channel version of Chrome and followed instructions carefully (including the command line switch), but must have missed something.  I’ll give this a closer look along with the many other features later.

March for Life

“I have set before you life and death, blessings and curses. Now choose life, so that you and your children may live.”
Deut 30:19b

This year’s March for Life was well-attended; several thousand marched in Atlanta, with hundreds of thousands in similar marches in Washington, D.C., and at state capitols across the nation.  Tina took some thought-provoking photos that you can view in her Picasa gallery.

Abortion is a moral issue, not a political one.  It is the slavery and holocaust of our day, defended by the same arguments, to the same end.  May it soon be just as far behind us.

Alpharetta Eagle Run 5K

This morning’s 2nd Annual Eagle Run 5K was an overcast, windy one, and the headwinds and moderate hills kept it interesting.  At 25:52, I was more than two minutes slower than my goal and training runs, but I’ll keep at it.  It benefited Milton High School boy’s lacrosse.  It seemed most of the team ran it, along with about 300 others.

Just Add Water

I decided to go with Ubuntu 9.10 64-bit this time on my newest development VM.  But when first starting the DB2 9.7 install (db2setup), I got the following errors:

WARNING:
The 32 bit library file libstdc++.so.6 is not found on the system.
32-bit applications may be affected.
ERROR:
The required library file libaio.so.1 is not found on the system.

Those are the standard C++ and asynchronous I/O libraries.  My fingers were already a bit tired from sudo apt-get install‘ing and config’ing the world, but a couple more (with !s:s/… shortcuts) wouldn’t kill me, so:

sudo apt-get install libstdc++6
sudo apt-get install libaio-dev

… and I was off and running.

One of these days, someone will package an “everything you could possibly every need” distro to speed setup.  Why not, disk and bandwidth are cheap.  Until then, maybe I’ll script the whole thing or at least save off my baseline VM.

The Runstats Secret to a Full Night’s Sleep

I’ve read some plausible explanations for why a DB2 runstats isn’t absolutely necessary following a reorg, but I’m not convinced.  Nearly all agree that runstats before (or as part of) a reorgchk is often wise, but some say it isn’t necessary after the reorg itself.  Sometimes the “evidence” centers around static tables that weren’t terribly fragmented in the first place, or even things like MDCs where reorgs aren’t needed anyway.

A call came in early this morning from a customer who had reorg’ed some very large tables over the long weekend but did not update statistics afterward.  They were greeted after the holiday with hung SQLs against one particular table, and had to page-awaken the DBA for authorized access.  In this case, the problem was corruption, which affected runstats itself.  That’s a nice side benefit: if a reorg resulted in index or table corruption, a good runstats (with distribution statistics) will often so fully cover the table that it’ll flush it out.  Runstats during the weekend maintenance would have identified the problem early and avoided production impacts later.

Runstats is usually so quick and easy that I come down on the side of pragmatism.  Who cares if theoretically it might not be necessary?  Do it for your health.  Or at least for a few extra hours sleep.

Let My Smalltalk Go

At some point while building etoy and Croquet pyramids in Egypt, Seaside happened.  But the Squeak children grumbled amongst themselves rather than pursue this new promised land with fixed purpose.  Aliens who just wanted that milk and honey (Seaside) had enough on their hands learning Hebrew (ahem, Smalltalk) to be bothered  by Squeak traditions.  So the sea parted and a new Smalltalk platform arose, taking Seaside contributors with it.

Pharo [sic] is a Smalltalk platform built from Squeak.  It’s smaller and cleaner than Squeak, and is replacing it as the Seaside base in that lineage.  The development and licensing model is very deliberately a benevolent dictatorship in order to focus efforts and push the community forward more quickly and efficiently.  Perhaps democracy is overrated.

My first reaction upon downloading and using Pharo is that it’s “Squeak with a haircut.”  The look and feel is cleaner (Mac-like), and there are no bright colors and etoys.  By default, it retains Squeak’s “one OS window” presentation, but that hardly matters when you just want to do Smalltalk and use it as a Seaside platform.  The base class libraries are very familiar, and it comes with all the expected browsers and tools, like the SUnit TestRunner and Monticello source control.

Loading Seaside is just one ScriptLoader loadSeaside30 away.  It gets even easier than that: the Seaside One-Click Experience is migrating from Squeak to Pharo.  That’s further evidence of Seaside’s new emphasis on Pharo over Squeak.

For a 1.0 RC 1, it’s quite stable and workable.  I did get several tool walkbacks, but nothing that I couldn’t work around and occasionally debug.  I’ll definitely keep an eye on this as it continues marching to the promised land.

Hands & Feet Project

“…to the extent that you did it to one of these brothers of Mine, even the least of them, you did it to Me.”
Matthew 25:40b

The scope of the damage, the length of relief, and the need for prayer are all so great right now in Haiti; it just doesn’t fit into the brain.  It could take decades to recover, but basic survival and rescue are the needs of the day right now.

When giving, it’s prudent to find a proven, reputable, right-minded organization that’s already there.  I commend to you the Hands & Feet Project, an orphanage in underserved Jacmel, Haiti.  Please visit their web site for more information.

Football Primary Sources

You can’t confuse information with insight.  There’s been much speculated and said about Georgia Tech’s NFL departures (Bay-Bay, Dwyer, Derrick Morgan, Burnett), but no news from those left to carry the banner.  So it was good to read OT Nick Claytor’s guest blog today.  Stop it with the analysis of the analysis, let’s hear it from the players.  Check it out: http://blogs.ajc.com/georgia-tech-sports/2010/01/14/special-guest-blog-from-nick-claytor-2/.

It’s an annual challenge for my alma mater to stay in the top of the FBS big leagues.  GT is very deliberately a small school (13,000 undergrads) with tough academic requirements, making recruiting a real challenge.  Of course, that’s why the triple option was so welcome.  If you can’t beat ’em on bulk, blind ’em with science.  Paul Johnson has lost some powerful weapons, but he’ll adjust, and will continue to coach players to new heights.

This fan and alum is thankful for an exciting season.  I’ll watch these veterans in the NFL and mark my calendar for Sep 4.

Cool Runnings

The weather was better than I expected (above freezing), but the hills were worse.  This morning’s Etowah Soccer 5K was a hilly run, slowing me down to below my normal 10K pace.  I enjoyed the challenge, but didn’t like ending with my record worst time.  Oh well, chalk it up as a training run.  The entertainment value was good, though: the comedy of hearing so many of the Etowah High students cross the finish shouting “those hills were killer!”

It was close to home and for a great cause, so I’ll consider it a challenge for next year.

Materialize This

One of my core incompetencies is photography: put me behind the lens and the results are consistently pathetic.  So I leave that to folks who know what they’re doing, like my wife.  I’m often impressed by how much preparation, selection, skill and patience is required leading up to that decisive moment when a scene becomes artfully frozen in time.  We never see that effort when viewing a well-done photograph, but we certainly appreciate it.

Often in large databases, some of the best and most heavily-accessed information is very expensive to derive.  Sales trends, productivity stats, workflow metrics, and the like require cranking through very large volumes of data, often joining many tables.  And it seems everyone wants to know the results, so these questions get asked far too often for slow queries to keep up.

Views on Ice

In DB2, this is where materialized query tables (MQTs) come in handy.  Known as summary tables in other contexts, MQTs are query results frozen in time.  Think views on ice.  You define a table built from a query and tell DB2 when and how to refresh its contents.  You can then query an MQT directly or instruct the optimizer to automatically access it instead of base tables.  This lets you to break the link between calculating the results and accessing them.  You can read all about them (including the related create table as and refresh table statements) in the DB2 documentation.

In DB2, there are two different ways to use MQTs and two different ways to refresh them.

Using MQTs

DB2 can automatically rewrite queries to use MQTs.  This is tempting when trying to tune a system without changing its SQL.  You must specify enable query optimization, populate and runstats the MQT early, update the current refresh age and current maintained table types for optimization registers, and toss a few dead chickens in the air.  Even then you may be surprised to find that DB2 won’t use your MQT.  You can run EXPLAINs to check, and messages in the db2exfmt output will tell you if the registers aren’t set correctly.  Yes, getting DB2 to rewrite to your MQT can be difficult.  Directly accessing the MQT in your SQLs is a surer route.

To directly access the MQT, simply write your SQLs against it.  In this case, you’re treating the MQT just like any other table, but you’re getting the speed and convenience of the refresh table command, as compared to manually updating a table via delete then select/insert or other means.

Refreshing MQTs

The two basic ways to refresh MQTs are immediate and deferred.  If you specify refresh immediate, DB2 automatically updates the MQT whenever the data feeding it changes.  This can significantly impact delete, update, and insert statements.  Refresh deferred means you will update the MQT according to your own timing, and is often the safest route.

Indexing

You can create indexes on MQTs and that’s often the point: you’ll likely find new wonderful index opportunities with persistent summary data.  An MQT without indexes is like a ship without a sail.  Or something like that.

Conclusion

MQTs are certainly not for every situation: most queries in OLTP systems don’t need them, and often results must be current up to the nanosecond.  But a few well-placed MQTs over time have yielded for me dramatic performance improvements.  So MQTs are definitely a tool to keep in your utility belt.   Or camera bag.

Needles in a Haystack

I recently posted snippets for finding and formatting DB2 event monitor files for a deadlock.  OK, so now what?  Once you get these often voluminous outputs, how do you quickly get to the bottom of it?  That question came up this evening.

Sometimes just grepping out the SQLs will reveal enough about what the offending application was doing to trigger a good aha moment.  That can be done simply with:

grep “Text” dl1.txt

Or, if in Windows without grep or cygwin:

find “Text” dl1.txt

Often, though, a little more context is needed.  If there are multiple deadlocks in the file, adding the Deadlock Event headers will separate them.  Adding the participant IDs will show who did what and how the SQLs are crossed.  And adding line numbers will provide places to jump to when opening the file in an editor to drill in further.  That command is quick and easy:

grep -n -e “Text” -e “Deadlock Event” -e “Participant no.” dl1.txt

And, of course, toss it to a file if needed for more convenient viewing:

grep -n -e “Text” -e “Deadlock Event” -e “Participant no.” dl1.txt > dl1-out.txt

Happy hunting!

Cancel or Allow?

“You are coming to a sad realization – Cancel or Allow?”

UAC talking to PC, Apple Ad

It’s not difficult, just tedious.  Getting modern Windows desktop software to run without any prompts from the UAC mall cop requires jumping through a few hoops, mainly attaching an application manifest and digital signature to the EXE.  Stop short of this and, depending on GPOs and other settings, users will either be unable to run the app, or will see annoying prompts when starting it.  This applies to all the UAC platforms: Windows 7, Windows Server 2008 (R1 and R2), and Vista.

Working with Jack today, we walked through this process for one of our client applications.  Fortunately, there are many tools and recipes available, including the excellent CodeProject article on Making Your Application UAC Aware and Microsoft’s own verbose Windows 7 Application Compatibility Testing Guide.

Microsoft’s MT tool attaches the manifest to the EXE, with hashing to protect against modification.  Famed Mark Russinovich provides the Sigcheck tool for dumping/verifying this.

It takes a few bucks to get a legitimate code-signing certificate (one that has been signed/vouched for by one of the root CA’s like Comodo), but you can create your own self-signed cert for testing purposes with MakeCert, add it to your local cert store with CertMgr, and then sign the EXE with SignTool.  Since a self-signed cert doesn’t have a known root CA covering it, you’ll have to import the cert into the local certificate store of any place you deploy it.  To avoid that, we’ll cough up the cash for a real cert before we ship.

Ultimately what tripped us up was the restriction that compliant apps shouldn’t write to their installation (Program Files) folder, or any folder beneath it.  We were aware of this, but a red herring message threw us off.  To get the bottom of it, we set the EXE to elevate to administrator (requestedExecutionLevel level=”requireAdministrator”) and verified that resolved the problem.  This narrowed things down and so we could find and fix where we were breaking the virtual store rules, and then return to the friendlier asInvoker level.  This finally silenced the mall cop.

Room to Grow

It’s a common pattern in many of the DB2 transaction systems I work with: periodically loading new partially-populated data into tables, and then continually adding data to these rows throughout the day.  If this additional data is stored in varchars or other variable-length types, the updated row stretches to a new, larger size.  Too much of this can lead to excessive page reorgs and overflows.  This is one of many things I look for when monitoring a new system, and the problem popped up yesterday when helping an internal group with their new database components.

Page reorgs are shuffling rows within a page (de-fragging) to make room for a now-larger row.  Overflows are having to split a page to two because there’s not enough free space in the current page to accommodate the updated row, even after reorg.  Too much of this can harm performance by increasing logical and physical I/Os.  Overflows are the most severe because DB2 must write and read two pages instead of one for a given row’s data.  Table snapshots report just how much of this is going on; the rate of change should be monitored after reorg’ing all tables.

One solution is to “prefill” or “pad” column data when initially inserting rows.  But this can be inefficient and is often impractical for systems with truly variable/dynamic data.

An easier solution is to increase PCTFREE, to allocate more free space in table and index pages, leaving room to grow.  When reorg’ing, DB2 packs rows into pages leaving at least that much space unused before jumping to the next page.   Finding the right PCTFREE number can be done by estimating the approximate percentage row growth and then incrementally increasing toward that number and measuring the impact.

There are two important (and often misunderstood) points to remember when changing PCTFREE:

  1. It does not take effect until you reorg the table.
  2. It does not apply to newly-inserted rows (after the reorg).  However, increasing PCTFREE and reorg’ing will leave more room on the page that will help afterward.

DB2s new row compression and multi-dimensional clustering (MDC) can, in the right situations, also help reduce page reorgs and overflows.  More on these later.

You Say Acquirer, I Say ODFI

I’ve been learning the details of Visa disputes processes and Visa Resolve Online (VROL); interesting stuff.  What strikes me each time I learn a new payment exceptions and returns workflow is just how similar they all are.  In many ways this is expected.  For example, NACHA and ECCHO built our ACH network from the old Electronic Check Presentment (ECP) systems, so it shouldn’t surprise us that it’s very “check-like,” complete with cash letters, batches, MICR data, overnight clearing, and very similar data interchange formats: ECP, ACH, X9.37, and X9.100-187.

But the Visa network didn’t have this legacy, so I’m surprised to find just how very conceptually similar it is to the check/image and ACH world.  Sometimes it looks like only the names were changed to protect the guilty.

So here’s my brief thesarus to translate Visa-speak to ACH-speak and check-speak:

Issuer = RDFI = Paying Bank

Acquirer = ODFI = Collecting Bank = (usually) BOFD

With these translations in hand, you can simply relabel the boxes in those issuer disputes and acquirer disputes diagrams and workflows, and you have classic check and ACH exceptions processes.  Voila!   Just relabel some screens and web pages and you can use a check or ACH system for Visa, right?

If only it were that simple.  There are clear technology and interface differences that keep the twain apart, yet consolidating these payment types is still very doable.  Perhaps the bigger issues are the organizational ones.  There have always been walls between the check/image, ACH, wire, and card departments, and those staffing/people issues are often the toughest to overcome.  But as payment trends continue to shift, clearly we’ll see more of that.

BTW, my little simplification assumes debit pull transactions.  That’s all check and card have, but ACH has credit push transactions, and these can be disputed and returned.  Do credit push exceptions/returns mean you change the workflow, or simply post and settle differently?  I’ve only done debit pulls, so I don’t know yet.

Treadmill Weather

I vaguely remember rolling over to check the weather on my Droid this morning around 6:00 a.m., contemplating running the Frost Bite 5K over in Acworth.  But with a racetime forceast of 16 degrees with a 4 degree wind chill, along with snow and ice covered roads, I quickly got over it.  “Frost bite,” indeed.  I applaud the few dozen who ran it; I’m sure there were some PRs set – for the coldest race.

Conditions were fine later that afternoon indoors on the treadmill at the Y: 70 degrees and calm.  And, of course, watching the televised Jackets victory over #5 Duke made it all the better.   Lawal’s game alone pushed me to run 6 miles at my 5K pace.  Maybe one day someone will host a January race indoors – with treadmills and Cardio Theatre.  Sign me up for that one!

Test the Testing

One of many benefits of test-driven development (TDD) and automated unit and regression testing is that it provides additional defect-detection safety nets.  Development teams catch more of their own defects before delivery to test.  In my own recent experience (and that of others I’ve talked with), this is very valuable and very timely.

In so many shops, recent cost cutting and offshoring has diminished the quality of component and system testing.  Often this decline is staff related: reduced tester availability and increased turnover, even within offshore test groups.  Defect escapes are costly and embarrassing.  So what’s a developer to do?

We can borrow one groovy solution from the Three Dog Night days.  Those dusty pages of the 70s Harlan Mills papers describe defect seeding: deliberately adding defects to monitor defect discovery rates and coverage.  Also known as bebugging (with a “b”), defect seeding objectively measures the quality of the testing.  In short, it tests the testing.

Deliberately adding bugs seems to go against the grain: it’s a destructive activity, as is test itself.  Yet, properly measured and managed, it can be an important part of overall software quality engineering.  If a test team misses known, well-placed bugs, this is identified and corrective action can be taken.  That’s much better than shooting in the dark.  It can shed much-needed light when the test team is in a different location and direct oversight isn’t possible.

Much of that 70s literature on bebugging was wrapped up with somewhat extreme, academic concepts – like limited (statistical) testing and the idea that tools could automatically inject and remove defects or faults.  Much simpler approaches are possible; such as:

  1. Developers deliberately code important but non-fatal defects in new code they write, while at the same time writing xUnit test cases that will fail because of the injected defects.  The tools themselves provide the metrics, and this avoids the risk of forgetting about some seeded bugs when it’s time to remove them after the test iteration.
  2. After releasing a sprint or build to test, the development team continues to actively test the product themselves.  Any defects found by the developers are privately recorded and tracked to determine if and when the test team finds them.

Simple bebugging practices such as these and others can, over time, improve test quality and coverage.  And you’ll have real metrics to see how well Test passes The Test.