WriteStreams of Consciousness

A friend and fellow Droidian asked me today what my favorite free Android apps were.  I really didn’t know where to begin because I use so many of them so regularly.  This includes, of course, the preloaded ones: music, calendar, gmail, browser, maps, navigation, talk, contacts, messaging, YouTube, Picasa, etc., etc..  But to answer the question, I thought I’d jot down a list of some of my favorite free Market downloads here:

- Scoreboard – All my sports scores, all the time.
- Pandora and Slacker – Personalized internet radio, love it.
- Google Places – Must… find… coffee.  Perfect GPS compliment to Maps and Navigation.
- Flixster Movies – What’s on and what will be on.
- Weather Channel – Faster than looking out the window.
- Google Sky Map, Clear Sky Droid – For my astronomy fixes.
- Google Goggles – Still under development, but fun to try.
- androidVNC and ConnectBot (ssh) – The remote access classics.
- Shop Savvy – Barcodes R us.
- NewsRob, wpToGo, Google Listen, Read Later – For feeds, blogs, podcasts, and news.
- KeePassDroid – The world’s most portable password safe.
- Evernote – Snap a photo or jot a note, and it’s on the web in seconds.
- “Commercial news” readers: Thomson Reuters, BBC News, USA Today, Fox News, etc.
- Toys and Games – Compass, Labyrinth, DailyStrip, Jewels, Sudoku, Solitaire, Frozen Bubble, etc., etc.
- And, of course, lots of mainly pointless apps like Run, Zombie, Run and Talking Parrot.

Of course, half the fun is searching Android Market for any app you can think of that just might be out there.  There’s a good chance it is.

Perhaps it’s that small bit of Irish blood in me, or simply that all my memories of live bagpipes are very pleasant ones, but good Irish music always brings good cheer.  Irish pipe band Ceol na Gael (“music of the Gaels”) from Charlotte, NC is especially skilled at it, and they continue to grow in fame and repertoire.

One of the band members, Wayne, is a friend and co-worker: a talented programmer and piper.  Check them out at their web site or Facebook page.  And if you’re a Carolinian in need of a good Irish fix, contact them.

The sheer number of DB2′s memory-related parameters can hit like a tidal wave.  Bufferpools, sort heaps, lock lists, package/statement caches, and various other heaps can all harm performance if undersized, and can waste or overcommit memory if oversized.  Further, many of the configuration parameters are interrelated; for example, increase logbufsz and you may also have to increase dbheap to accomodate it.  DBAs and performance tuners can feel overwhelmed by the number of knobs one must turn in each database.  And developers can feel overburdened when providing scripts and configuration tools to right-size their databases for each customer’s unique volume requirements.

Fortunately the latest (9.x) versions of DB2 LUW provide the Self-Tuning Memory Manager (STMM).  This feature significantly simplifies memory management in DB2, letting the database automatically size memory areas based on activity and needs.  For most parameters, you can also specify an optional initial size to avoid the overhead of progressively stretching memory after startup.

To turn on STMM for a database, specify:

update db cfg using self_tuning_mem on

You’ll likely also want database_memory set to automatic:

update db cfg using database_memory automatic

You can fix-size or initial-size database_memory, or use the computed option for back-compatibility.  The publib DB2 documentation has more information.

At this point, you can auto-size many of the memory areas.  For example, for dynamic lock list sizing, specify:

update db cfg using maxlocks automatic
update db cfg using locklist automatic

Depending on DB2 version, you can configure any of the following this way: dbheap, pckcachesz, sortheap, stmtheap, applheapsz, appl_memory, stat_heap_sz, dft_prefetch_sz, num_iocleaners, num_ioservers, maxappls, and avg_appls.   Anyone who has tried to package or build scripts that set these beyond the old (painfully too low) defaults can appreciate the benefits of “going automatic.”

Perhaps the greatest benefit comes when auto-sizing bufferpools.  For that, specify:

alter bufferpool <poolname> size automatic

To specify an initial (startup) size (in pages), place it before the automatic keyword.

In my experience so far, “going automatic” has worked swimmingly, with one exception: sort heaps.  OLTP systems can, at times, experience heavy loads of small sorts or sort-like activity (hash joins, grpby’s, etc.).  It’s not uncommon for too many of these to overflow (spill), or for a sub-optimal access plan to arise, even when more than enough memory is available for sorts.  This is partly because auto-sizing the sort heap requires some extra work and partly because the optimizer can’t predict future sizes.  More on this in a future post.

I no longer actively use Google Web Toolkit (GWT), but some affinity remains.  I don’t miss experiencing its growing pains from 1.3 to 1.6: maintaining parallel hierarchies of cross-compilable business and transfer objects, forging and tweaking copy and mutation frameworks, debugging generated JavaScript, deployment issues, waiting seemingly forever for Java 5 and 6 support, and so on.  But spending trench time in any new technology has a way of fostering interest in seeing it mature.

And mature it has!

After a long stream of GWT updates with relatively small incremental enhancements, GWT 2.0 is a major leap forward, and a game changer.  It’s the GWT I’ve always wanted, and one I wish I had at the beginning.

Google has finally taken on the tools side of the equation with their own Eclipse plug-in.  Its operation is smooth and seamless: no more cobbled-up mash of third-party plugins, Cypal studio, and gwtCompile Ant tasks.

I’m thrilled to see the death of hosted mode; Google now offers native browser plug-ins for debugging.  I used the plug-ins for Firefox (3.5.7) and Chrome (4.0.302.3, dev channel) and debugging was a breeze, albeit sluggish at times.  Among other things, this ends the problem of code working one way in hosted mode and another way (or not at all) when deployed to a browser.

The Speed Tracer looks promising for performance monitoring and tuning.  However, I could not get past its “no data received in 6 seconds” error.  I used the latest dev channel version of Chrome and followed instructions carefully (including the command line switch), but must have missed something.  I’ll give this a closer look along with the many other features later.

Buried for nearly three days in the rubble of the Haitian earthquake, Dan Woolley of Compassion International put his iPhone to good use.  Read about it at msnbc.com.

“I have set before you life and death, blessings and curses. Now choose life, so that you and your children may live.”
Deut 30:19b

This year’s March for Life was well-attended; several thousand marched in Atlanta, with hundreds of thousands in similar marches in Washington, D.C., and at state capitols across the nation.  Tina took some thought-provoking photos that you can view in her Picasa gallery.

Abortion is a moral issue, not a political one.  It is the slavery and holocaust of our day, defended by the same arguments, to the same end.  May it soon be just as far behind us.

This morning’s 2nd Annual Eagle Run 5K was an overcast, windy one, and the headwinds and moderate hills kept it interesting.  At 25:52, I was more than two minutes slower than my goal and training runs, but I’ll keep at it.  It benefited Milton High School boy’s lacrosse.  It seemed most of the team ran it, along with about 300 others.

March will be a busy month in the electronic payments world, especially for returns.  New rules and standards kick in for both ACH and image exchange networks.  These changes go out of their way for back-compatibility, so it should all go smoothly.

Image Exchange

The wonderful thing about standards is there are so many to choose from.  Shortcomings in ANSI X9.37 led to a proliferation of “companion documents” to correct and fill in the gaps: there was a different companion document for each major exchange.  X9.100-180 tried to modernize X9.37 and cover everyone’s wish lists, but it was too great a change for anyone to adopt (the “180″ suffix was fitting).  So the more evolutionary X9.100-187 took its place and the good folks at CheckImage Central consolidated companion docs into Universal Companion Documents (UCDs) – first for X9.37, then for X9.100-187.  Taken together, you have a complete picture: X9 defines the standard, and the UCD describes how to apply it.  Universal standards adoption: a good thing.

March is the deadline for banks and exchanges to receive files for the first X9.100-187 + UCD version.   Those who support X9.37 and its UCD are already there, in compatibility mode.  But with a little ambition one can move beyond mere compatibility, prepare for September (the send/’30′ deadline), and start taking advantage of the new fields, including:

ACH

NACHA’s ACH Rules Amendment #2-2009 kicks in on March 19.  Most of the changes are “outside the network” (manual), such as relaxing the rules about the former WSUPP (pronounced “whassup”) form for unauthorizeds, and aligning Stop Pays with Reg E.  But R39 (improper source document) can now also be used for duplicate transaction returns.  And so it seems the climate lately emphasizes back-compatibility over capability.  Think Windows 7 and Vista.

Schizophrenia

Perhaps that’s why both the X9 and ACH changes suffer from return reason schizophrenia.  ANSI and NACHA each added new meanings to existing return reason codes, avoiding the software/profile updates that new codes require.  But this splashing around to avoid making ripples could create bigger waves in the end.

ANSI return codes “I” and “Q” now have different meanings based on return file type – admin or monetary.  Because admins occasionally arrive in monetary return files, ambiguity is possible.  An “I” on a monetary return has always meant “endorsement missing” but now it can mean “image missing” for an admin return.  That’s a stroke of bad luck, since some systems can automatically re-present endorsement missing/guarantee items (a big time saver), but re-presenting without an image is a bad thing.   Distinct return codes would have been nice, but even a lower-level (item-level, not file level) admin return indicator would have covered this and allowed more flexibility.  So folks will have to to split admins out into separate files through eternity.

I follow NACHA’s logic for re-using R39 (improper source document) for duplicate check returns.  But since each points to a very different error (and culprit) in a check conversion process, this ambiguity makes it more difficult to monitor and correct things.  And since forward presentment check conversion is a high-volume activity, good data to support continual process improvement is important.

The Next Step

Like legislation, standards are rarely edited or repealed; rather, new ones get layered atop them.   These changes are in place, but opportunity remains for further improvements.  I’ve alluded to more item-level control and a forward-presentment (25 record) version of 31.14.  More on these later.

I decided to go with Ubuntu 9.10 64-bit this time on my newest development VM.  But when first starting the DB2 9.7 install (db2setup), I got the following errors:

WARNING:
The 32 bit library file libstdc++.so.6 is not found on the system.
32-bit applications may be affected.
ERROR:
The required library file libaio.so.1 is not found on the system.

Those are the standard C++ and asynchronous I/O libraries.  My fingers were already a bit tired from sudo apt-get install‘ing and config’ing the world, but a couple more (with !s:s/… shortcuts) wouldn’t kill me, so:

sudo apt-get install libstdc++6
sudo apt-get install libaio-dev

… and I was off and running.

One of these days, someone will package an “everything you could possibly every need” distro to speed setup.  Why not, disk and bandwidth are cheap.  Until then, maybe I’ll script the whole thing or at least save off my baseline VM.

I’ve read some plausible explanations for why a DB2 runstats isn’t absolutely necessary following a reorg, but I’m not convinced.  Nearly all agree that runstats before (or as part of) a reorgchk is often wise, but some say it isn’t necessary after the reorg itself.  Sometimes the “evidence” centers around static tables that weren’t terribly fragmented in the first place, or even things like MDCs where reorgs aren’t needed anyway.

A call came in early this morning from a customer who had reorg’ed some very large tables over the long weekend but did not update statistics afterward.  They were greeted after the holiday with hung SQLs against one particular table, and had to page-awaken the DBA for authorized access.  In this case, the problem was corruption, which affected runstats itself.  That’s a nice side benefit: if a reorg resulted in index or table corruption, a good runstats (with distribution statistics) will often so fully cover the table that it’ll flush it out.  Runstats during the weekend maintenance would have identified the problem early and avoided production impacts later.

Runstats is usually so quick and easy that I come down on the side of pragmatism.  Who cares if theoretically it might not be necessary?  Do it for your health.  Or at least for a few extra hours sleep.

At some point while building etoy and Croquet pyramids in Egypt, Seaside happened.  But the Squeak children grumbled amongst themselves rather than pursue this new promised land with fixed purpose.  Aliens who just wanted that milk and honey (Seaside) had enough on their hands learning Hebrew (ahem, Smalltalk) to be bothered  by Squeak traditions.  So the sea parted and a new Smalltalk platform arose, taking Seaside contributors with it.

Pharo [sic] is a Smalltalk platform built from Squeak.  It’s smaller and cleaner than Squeak, and is replacing it as the Seaside base in that lineage.  The development and licensing model is very deliberately a benevolent dictatorship in order to focus efforts and push the community forward more quickly and efficiently.  Perhaps democracy is overrated.

My first reaction upon downloading and using Pharo is that it’s “Squeak with a haircut.”  The look and feel is cleaner (Mac-like), and there are no bright colors and etoys.  By default, it retains Squeak’s “one OS window” presentation, but that hardly matters when you just want to do Smalltalk and use it as a Seaside platform.  The base class libraries are very familiar, and it comes with all the expected browsers and tools, like the SUnit TestRunner and Monticello source control.

Loading Seaside is just one ScriptLoader loadSeaside30 away.  It gets even easier than that: the Seaside One-Click Experience is migrating from Squeak to Pharo.  That’s further evidence of Seaside’s new emphasis on Pharo over Squeak.

For a 1.0 RC 1, it’s quite stable and workable.  I did get several tool walkbacks, but nothing that I couldn’t work around and occasionally debug.  I’ll definitely keep an eye on this as it continues marching to the promised land.

“…to the extent that you did it to one of these brothers of Mine, even the least of them, you did it to Me.”
Matthew 25:40b

The scope of the damage, the length of relief, and the need for prayer are all so great right now in Haiti; it just doesn’t fit into the brain.  It could take decades to recover, but basic survival and rescue are the needs of the day right now.

When giving, it’s prudent to find a proven, reputable, right-minded organization that’s already there.  I commend to you the Hands & Feet Project, an orphanage in underserved Jacmel, Haiti.  Please visit their web site for more information.

You can’t confuse information with insight.  There’s been much speculated and said about Georgia Tech’s NFL departures (Bay-Bay, Dwyer, Derrick Morgan, Burnett), but no news from those left to carry the banner.  So it was good to read OT Nick Claytor’s guest blog today.  Stop it with the analysis of the analysis, let’s hear it from the players.  Check it out: http://blogs.ajc.com/georgia-tech-sports/2010/01/14/special-guest-blog-from-nick-claytor-2/.

It’s an annual challenge for my alma mater to stay in the top of the FBS big leagues.  GT is very deliberately a small school (13,000 undergrads) with tough academic requirements, making recruiting a real challenge.  Of course, that’s why the triple option was so welcome.  If you can’t beat ‘em on bulk, blind ‘em with science.  Paul Johnson has lost some powerful weapons, but he’ll adjust, and will continue to coach players to new heights.

This fan and alum is thankful for an exciting season.  I’ll watch these veterans in the NFL and mark my calendar for Sep 4.

The weather was better than I expected (above freezing), but the hills were worse.  This morning’s Etowah Soccer 5K was a hilly run, slowing me down to below my normal 10K pace.  I enjoyed the challenge, but didn’t like ending with my record worst time.  Oh well, chalk it up as a training run.  The entertainment value was good, though: the comedy of hearing so many of the Etowah High students cross the finish shouting “those hills were killer!”

It was close to home and for a great cause, so I’ll consider it a challenge for next year.

One of my core incompetencies is photography: put me behind the lens and the results are consistently pathetic.  So I leave that to folks who know what they’re doing, like my wife.  I’m often impressed by how much preparation, selection, skill and patience is required leading up to that decisive moment when a scene becomes artfully frozen in time.  We never see that effort when viewing a well-done photograph, but we certainly appreciate it.

Often in large databases, some of the best and most heavily-accessed information is very expensive to derive.  Sales trends, productivity stats, workflow metrics, and the like require cranking through very large volumes of data, often joining many tables.  And it seems everyone wants to know the results, so these questions get asked far too often for slow queries to keep up.

Views on Ice

In DB2, this is where materialized query tables (MQTs) come in handy.  Known as summary tables in other contexts, MQTs are query results frozen in time.  Think views on ice.  You define a table built from a query and tell DB2 when and how to refresh its contents.  You can then query an MQT directly or instruct the optimizer to automatically access it instead of base tables.  This lets you to break the link between calculating the results and accessing them.  You can read all about them (including the related create table as and refresh table statements) in the DB2 documentation.

In DB2, there are two different ways to use MQTs and two different ways to refresh them.

Using MQTs

DB2 can automatically rewrite queries to use MQTs.  This is tempting when trying to tune a system without changing its SQL.  You must specify enable query optimization, populate and runstats the MQT early, update the current refresh age and current maintained table types for optimization registers, and toss a few dead chickens in the air.  Even then you may be surprised to find that DB2 won’t use your MQT.  You can run EXPLAINs to check, and messages in the db2exfmt output will tell you if the registers aren’t set correctly.  Yes, getting DB2 to rewrite to your MQT can be difficult.  Directly accessing the MQT in your SQLs is a surer route.

To directly access the MQT, simply write your SQLs against it.  In this case, you’re treating the MQT just like any other table, but you’re getting the speed and convenience of the refresh table command, as compared to manually updating a table via delete then select/insert or other means.

Refreshing MQTs

The two basic ways to refresh MQTs are immediate and deferred.  If you specify refresh immediate, DB2 automatically updates the MQT whenever the data feeding it changes.  This can significantly impact delete, update, and insert statements.  Refresh deferred means you will update the MQT according to your own timing, and is often the safest route.

Indexing

You can create indexes on MQTs and that’s often the point: you’ll likely find new wonderful index opportunities with persistent summary data.  An MQT without indexes is like a ship without a sail.  Or something like that.

Conclusion

MQTs are certainly not for every situation: most queries in OLTP systems don’t need them, and often results must be current up to the nanosecond.  But a few well-placed MQTs over time have yielded for me dramatic performance improvements.  So MQTs are definitely a tool to keep in your utility belt.   Or camera bag.

I recently posted snippets for finding and formatting DB2 event monitor files for a deadlock.  OK, so now what?  Once you get these often voluminous outputs, how do you quickly get to the bottom of it?  That question came up this evening.

Sometimes just grepping out the SQLs will reveal enough about what the offending application was doing to trigger a good aha moment.  That can be done simply with:

grep “Text” dl1.txt

Or, if in Windows without grep or cygwin:

find “Text” dl1.txt

Often, though, a little more context is needed.  If there are multiple deadlocks in the file, adding the Deadlock Event headers will separate them.  Adding the participant IDs will show who did what and how the SQLs are crossed.  And adding line numbers will provide places to jump to when opening the file in an editor to drill in further.  That command is quick and easy:

grep -n -e “Text” -e “Deadlock Event” -e “Participant no.” dl1.txt

And, of course, toss it to a file if needed for more convenient viewing:

grep -n -e “Text” -e “Deadlock Event” -e “Participant no.” dl1.txt > dl1-out.txt

Happy hunting!

“You are coming to a sad realization – Cancel or Allow?”

UAC talking to PC, Apple Ad

It’s not difficult, just tedious.  Getting modern Windows desktop software to run without any prompts from the UAC mall cop requires jumping through a few hoops, mainly attaching an application manifest and digital signature to the EXE.  Stop short of this and, depending on GPOs and other settings, users will either be unable to run the app, or will see annoying prompts when starting it.  This applies to all the UAC platforms: Windows 7, Windows Server 2008 (R1 and R2), and Vista.

Working with Jack today, we walked through this process for one of our client applications.  Fortunately, there are many tools and recipes available, including the excellent CodeProject article on Making Your Application UAC Aware and Microsoft’s own verbose Windows 7 Application Compatibility Testing Guide.

Microsoft’s MT tool attaches the manifest to the EXE, with hashing to protect against modification.  Famed Mark Russinovich provides the Sigcheck tool for dumping/verifying this.

It takes a few bucks to get a legitimate code-signing certificate (one that has been signed/vouched for by one of the root CA’s like Comodo), but you can create your own self-signed cert for testing purposes with MakeCert, add it to your local cert store with CertMgr, and then sign the EXE with SignTool.  Since a self-signed cert doesn’t have a known root CA covering it, you’ll have to import the cert into the local certificate store of any place you deploy it.  To avoid that, we’ll cough up the cash for a real cert before we ship.

Ultimately what tripped us up was the restriction that compliant apps shouldn’t write to their installation (Program Files) folder, or any folder beneath it.  We were aware of this, but a red herring message threw us off.  To get the bottom of it, we set the EXE to elevate to administrator (requestedExecutionLevel level=”requireAdministrator”) and verified that resolved the problem.  This narrowed things down and so we could find and fix where we were breaking the virtual store rules, and then return to the friendlier asInvoker level.  This finally silenced the mall cop.

It’s a common pattern in many of the DB2 transaction systems I work with: periodically loading new partially-populated data into tables, and then continually adding data to these rows throughout the day.  If this additional data is stored in varchars or other variable-length types, the updated row stretches to a new, larger size.  Too much of this can lead to excessive page reorgs and overflows.  This is one of many things I look for when monitoring a new system, and the problem popped up yesterday when helping an internal group with their new database components.

Page reorgs are shuffling rows within a page (de-fragging) to make room for a now-larger row.  Overflows are having to split a page to two because there’s not enough free space in the current page to accommodate the updated row, even after reorg.  Too much of this can harm performance by increasing logical and physical I/Os.  Overflows are the most severe because DB2 must write and read two pages instead of one for a given row’s data.  Table snapshots report just how much of this is going on; the rate of change should be monitored after reorg’ing all tables.

One solution is to “prefill” or “pad” column data when initially inserting rows.  But this can be inefficient and is often impractical for systems with truly variable/dynamic data.

An easier solution is to increase PCTFREE, to allocate more free space in table and index pages, leaving room to grow.  When reorg’ing, DB2 packs rows into pages leaving at least that much space unused before jumping to the next page.   Finding the right PCTFREE number can be done by estimating the approximate percentage row growth and then incrementally increasing toward that number and measuring the impact.

There are two important (and often misunderstood) points to remember when changing PCTFREE:

1. It does not take effect until you reorg the table.
2. It does not apply to newly-inserted rows (after the reorg).  However, increasing PCTFREE and reorg’ing will leave more room on the page that will help afterward.

DB2s new row compression and multi-dimensional clustering (MDC) can, in the right situations, also help reduce page reorgs and overflows.  More on these later.

I’ve been learning the details of Visa disputes processes and Visa Resolve Online (VROL); interesting stuff.  What strikes me each time I learn a new payment exceptions and returns workflow is just how similar they all are.  In many ways this is expected.  For example, NACHA and ECCHO built our ACH network from the old Electronic Check Presentment (ECP) systems, so it shouldn’t surprise us that it’s very “check-like,” complete with cash letters, batches, MICR data, overnight clearing, and very similar data interchange formats: ECP, ACH, X9.37, and X9.100-187.

But the Visa network didn’t have this legacy, so I’m surprised to find just how very conceptually similar it is to the check/image and ACH world.  Sometimes it looks like only the names were changed to protect the guilty.

So here’s my brief thesarus to translate Visa-speak to ACH-speak and check-speak:

Issuer = RDFI = Paying Bank

Acquirer = ODFI = Collecting Bank = (usually) BOFD

With these translations in hand, you can simply relabel the boxes in those issuer disputes and acquirer disputes diagrams and workflows, and you have classic check and ACH exceptions processes.  Voila!   Just relabel some screens and web pages and you can use a check or ACH system for Visa, right?

If only it were that simple.  There are clear technology and interface differences that keep the twain apart, yet consolidating these payment types is still very doable.  Perhaps the bigger issues are the organizational ones.  There have always been walls between the check/image, ACH, wire, and card departments, and those staffing/people issues are often the toughest to overcome.  But as payment trends continue to shift, clearly we’ll see more of that.

BTW, my little simplification assumes debit pull transactions.  That’s all check and card have, but ACH has credit push transactions, and these can be disputed and returned.  Do credit push exceptions/returns mean you change the workflow, or simply post and settle differently?  I’ve only done debit pulls, so I don’t know yet.

I vaguely remember rolling over to check the weather on my Droid this morning around 6:00 a.m., contemplating running the Frost Bite 5K over in Acworth.  But with a racetime forceast of 16 degrees with a 4 degree wind chill, along with snow and ice covered roads, I quickly got over it.  ”Frost bite,” indeed.  I applaud the few dozen who ran it; I’m sure there were some PRs set – for the coldest race.

Conditions were fine later that afternoon indoors on the treadmill at the Y: 70 degrees and calm.  And, of course, watching the televised Jackets victory over #5 Duke made it all the better.   Lawal’s game alone pushed me to run 6 miles at my 5K pace.  Maybe one day someone will host a January race indoors – with treadmills and Cardio Theatre.  Sign me up for that one!