Author Archives: Derek

About Derek

Derek Williams is a Senior Engineering Architect for Fiserv, Inc. Residing in the foothills near Canton, GA, Derek is a father of six, husband of one, coach of thirteen, and your servant. He can be reached at

Groovy Fast

I sometimes find myself trying out small, standalone Java snippets, much as I do in other languages. But unlike other languages, Java doesn’t provide a built-in REPL for this. Sure, I can wrap the snippets in JUnits or small main programs, or use Eclipse scrapbook pages, but that’s the slow way to experiment. I want something lightweight and fast for this decidedly heavyweight and slow language.

In the past, I’ve used different Java shells and sites to fill this gap, but since I’m working in Groovy more lately, my new favorite is the Groovy Console. And when I need snippets integrated with my code and environment, the Groovy-Eclipse plugin provides that.


I had to write an XML transform today that included CDATA, passed though exactly as received without encoding or escaping characters. It had been awhile, so I dug through an old project to find the incantation – ah: cdata-section-elements.  Just include that magic word in xsl:output, listing the elements to pass through, like so:

   cdata-section-elements="myPassThroughData" />

BTW, this tool is quite helpful when fiddling with XSLT fragments:

You Again

Remember back in the JDK 1.5 days when some genius changed BigDecimal.toString and broke half the universe? I’ve felt the impacts in DB2, Oracle, VoiceObjects, and other places. I’m over that now (and all the cool kids have updated or switched to toPlainString), but this thing just keeps coming back to haunt me.

Like this week, when old DB2 JDBC (JCC) drivers kept showing up on some platforms, prompting these intermittent errors:

[ibm][db2][jcc][converters] Overflow occurred during numeric data type conversion of …

This error can sneak up on ya’ because it often takes just the right numbers to kick in. Like the lottery, and sometimes with the same odds.

Fortunately, the fix is easy: just update JDBC drivers to something from the modern era.

Storyboards in any Language

Since the advent of NSLocalizedString, iOS localization practices haven’t changed much. But the new storyboards are easier to translate, particularly if you enable internationalization from the outset. To do this in Xcode 5:

  1. From the project settings (Info page), check Use base internationalization.
  2. For each storyboard, go to the File Inspector (utilities pane), enable Localization, and check at least Base and English.  This will extract strings files that others can translate or customize.
  3. Un-install and then re-install the app from target simulators or devices.

After initial setup, adding languages and dropping in strings files is easy, and quicker than ibtool-importing strings into individual xibs.

Short Form

A friend today challenged me on letting this blog go a bit dormant.  True, I once averaged about a post a day; now it’s about one a month.  The thing he missed most were the (searchable) tips and quick fixes that I often tuck away here.

So, I’m going to pick it up a bit, favoring short form over long form and frequency over duration, but hopefully not quantity over quality. It won’t be thing a day, but at least it’ll be better than silence. Hopefully.

Open for Repairs

Although Android Libraries are the recommend way to package and use larger components, there’s no clean way to handle them in a Maven-based Eclipse or Jenkins environment. Sure, JARs are easy enough, but not the co-requisite resource files. Checking these into source control is just wrong, and full Maven support for these libs in incomplete, even with m2e, m2e-android, and android-maven-plugin in place.

Apklibs provide a decent stopgap, so I settled on that approach: loading the zips (ahem, apklibs) to Nexus and creating scripts to link things up during build. It requires one manual step in Eclipse, but no biggie. Yet there are greener pastures ahead…

The upcoming new Android Build System addresses this problem with AARs. Of course, it goes much further, ditching Maven for Gradle, ADT for Android Studio, Eclipse for IntelliJ, and XML for Groovy. I like new things, but while we’re waiting on the new house, I wish Google would finish fixing the plumbing in the current one.


Code analyzers like FindBugs, PMD, and Sonar provide a nice safety net for catching potential quality issues early, during code and build.

By default, PMD over-emphasizes style, leading to a lower signal-to-noise ratio, but FindBugs is quite good at catching real issues that slip past standard, compiler, lint, and IDE checks. Surprisingly good at times, given that there’s only so much you can learn from static analysis.  So I was skeptical that any commercial static analyzer could “take it to the next level” and provide qualitative improvements over open source tools.

That is, until I tried Coverity.

I recently got the chance to try Coverity’s Advisors hands-on, and met with two of their sales engineers to learn more. Coverity is not cheap, but its cost can often easily be justified for commercial code based on time savings and quality improvements.

I ran Coverity against three projects: two server apps with UIs and web services, and an Android app. In all three cases, I was impressed with the findings, particularly the High and Medium Impact ones like resource leaks and the various null dereference checks. These are the value add checks that you don’t get from its embedded FindBugs or from other open source tools. Of course, the UI presentations like code paths (in both Eclipse and the web interface) were also very helpful.

There were quite a few false positives, but not enough to be a problem. Many of the issues were deliberate fail-fasts and mocks in JUnit test classes, and could be marked as ignores. Some of the more interesting finds were null dereferences of results from calls to third party libraries – where I expected a method to always throw an exception on error, but in some situations it could return null instead. This demonstrates how the tool “looks” into underlying JARs to find undocumented behaviors.

Coverity also provides a broad range of security advisor and webapp-security checks, looking for vulnerabilities to SQL injection, XSS, and other exploits in the OWASP and CWE lists. Coverity can receive feeds from other complementary analyzers such as Android Lint, clang, PMD, etc., and include them on its dashboards. This is a nice feature to manage all issues from one common database.

Full runs with most checks enabled took time, often over 30 minutes, so these wouldn’t be done on every continuous integration build. If Coverity ever adds incremental analysis, that would help there.

It’s impossible for developers to catch all potential weaknesses through manual code inspections alone, so when it comes to static quality analyzers, good tools are important. Coverity is top-notch, and this evaluation demonstrated to me the value of a purchased tool.

Droid Units

JUnit testing of model and service classes under Android/Dalvik is straightforward, but when it comes to testing UIs (activities and fragments), the basic support is just too low-level. Fortunately, Robotium (think Selenium for droids) provides a most-excellent adjunct for this.

Yet Robotium should be used with care to create maintainable tests that aren’t brittle. To that end, I’ve developed some UI automation practices:

  • Write true unit test classes that cover only a single activity or fragment. Of course, many UI actions will take you to the next activity (and you should assert that), but leave multi-activity testing for separate integration tests.
  • Stub back-end behaviors using standard mock and dependency injection techniques. For my current app, I kept it simple/lightweight and wrote my own code, but I’ve also used Mockito (with dexmaker) and Dagger (with javax.inject); these are nice Android-compatible frameworks.
  • Rather than repeating the same raw Robotium calls directly from your tests, wrap them in descriptive helper methods like enterUserID(String id), enterPassword(String password), clickLoginButton(), etc. This DRY approach makes for more readable tests and simplifies updates when your UI changes.
  • Since you probably use common superclasses for your activities and fragments, also create parent test case classes to factor common testing behaviors. See below for snippets from one of mine.

I haven’t found a good tool for measuring code coverage for apps (Emma under Android is flakely), so I’d love to hear your recommendations.

public abstract class BaseActivityTest<T extends Activity> 
                             extends ActivityInstrumentationTestCase2<T> {
	protected Solo mSolo;	
	public BaseActivityTest(Class<T> activityClass) {
	protected void setUp() throws Exception {
		// Reset your DI container and common mocks here...		
		mSolo = new Solo(getInstrumentation(), getActivity());
	protected void tearDown() throws Exception {
	public void testLayoutPortrait() {
	public void testLayoutLandscape() {
	// ...


There are several options for over-the-air distribution of in-progress apps to internal testers. For Android, you can start simple with a link to the APK on a web site, but this misses the benefits of the Google Play store: automatic updates, consistent store listings, crash reports, support links, etc. Fortunately, we now have Google’s new Private Channel Distribution: a protected area for distributing apps as they develop and get ready for the larger “real world” of public distribution.

The service is easy to use, but does require a few setup steps:

  1. You must be a Google Apps user with administrator access. If you don’t already have it, sign up and pay the man.
  2. You must be registered as a Google Play developer with an account belonging to your Google Apps domain. Here again, sign up and pay the man.
  3. Sign into your Google Apps Console and enable the Google Play Private Channel. The process changed again recently with the new admin console, but you can find the latest instructions here.
  4. Log in to the Google Play Developer Console and create, configure, and publish your app. Check the box under Pricing & Distribution – Restrict Distribution.

To install, add your Google Apps account to your phone (for example, Play Store – menu – settings –  Account – Add Account – OK – Existing).  You should then see the private apps for your Google Apps domain under Apps – Categories (this, too, has changed recently).

You can use the Recent changes section of the app listing to describe updates to testers, and include a link to a video walk-through. Perhaps soon WebEx’s Android app will provide mobile screen sharing for live demos; until then Android Screencast works well with WebEx PC desktop sharing.  It’s not as nice as Reflector, but it is free and lightweight.

Worth the Weight

With many Android apps, there often seems to be little correlation between the size of the package and the value it provides. Multi-megabyte apps that do almost nothing leave me wondering, “what’s in there?”

Unpacking with dex2jar and JD-GUI often provides answers, and frequently it just means the developer forgot to enable Proguard when building. An overabundance of ill-compressed drawables are another common source. But beyond these, the habits of server-side re-use (freely expanding POMs and dropping in FOSS JARs) are a key source of bloat.

I try to be stingy when it comes to Android app libs, often taking a tougher route to avoid bringing in large JARs that might otherwise be useful. Such was the case recently when I needed to do a multi-part post to a REST web service that consumed a mix of binary image data and JSON. Multipart HTTP is conceptually simple, but the markup is obscure enough to make generating it directly from a business app just wrong.

Fortunately, though, Apache HttpMime is just 26K, and makes the process simple. For example:

MultipartEntity entity = new MultipartEntity();
entity.addPart("request", new StringBody(request));
entity.addPart("image", new ByteArrayBody(image, "image/jpeg", filename));

To avoid duplicate class errors at Proguard time, exclude the dependent httpcore in your pom.xml, like so:


I don’t always add JARs to Android apps, but when I do, I prefer light ones.

Just Click

Distributing in-progress iOS apps is tedious enough: setting up signing certs, creating provisioning profiles, adding device IDs, etc. I wanted at least the end-user install step to be friendlier than syncing with iTunes and remembering the .mobileprovision file. That is, I wanted to provide the same simple install for my iOS apps as I do for Android apps, with a web page that folks can visit from their phone browser and simply click a link to download and install.

Fortunately, I came across the itms-services protocol: Apple’s solution for wireless app installs. All it takes is the distribution IPA (with embedded mobileprovision file), plist, and a web page with the link. It’s not widely known, but there are a couple of good tutorials on the web, such as Apple’s instructions and Aaron Perecki’s walkthrough. No more tangles, no more wires!

Out of the Box

Business travels and extra duties lately have brought some unexpected surprises. I find such “out of the box” adventures refreshing: pulling the rug out from under typical routines forces me to challenge assumptions and improvise. Here are some examples from the past two days.

The Right Sequence

During group architecture sessions, I needed to quickly create, modify and display sequence diagrams to demonstrate interactions among several new web services. Lacking my usual tools (and convinced there had to be a better mousetrap), I searched around and found This STTCPW site was exactly what I needed. Using it was faster than tools such as Visio Pro, and even faster than drawing and editing on whiteboards.

Weekend Jobs

Today I got word that some Quartz jobs weren’t running as expected during pre-production testing, where typical weekday processes were run as a logical business day on Saturday. The cron-expressions were correct, so I needed to see more. A look at the full jobs.xml revealed the culprit: this environment had been configured to use WeeklyCalendar rather than the usual AnnualCalendar. A quick temporary switch had Quartz working weekends with the rest of us.

Fog of WAR

Despite version labels and other indicators, a web app was behaving as if it was down-level in a key area. Since the WAR file hadn’t been obfuscated, I grabbed a copy of the JD-GUI decompiler for a closer look. Sure enough, the class in question was at an older level. When sources aren’t available or are in question, the JD Project tools are indispensable for moving past assumptions and getting to facts.

Decimal, with an E

New calls to my web services from VoiceObjects were sending certain decimal data in E notation.  The underlying call flow objects were TTA – Literals/Digits with a digits?minlength=1;maxlength=14 grammar, so it should have been decimal from end to end.  But there turned out to be an unexpected feature of the platform to convert to float whenever any arithmetic is done, with no access to DecimalFormat or other mechanisms to convert back. Since there was fortunately no loss of precision and since a problem correctly stated often solves itself, I knew exactly what to do: modify the web service to accept the format. Voxeo provides a great VoiceXML platform, but it has plenty of surprising nuances like this.

Move it, Move it

Today I had to move a few dozen DB2 LUW databases from one Windows system to another. I’ve been down this road before, and I always script the process, but this time I initially left out tablespaces. Bummer, the downside of writing db2look commands from memory.

So that I don’t get fooled again, I’m posting my simple scripts which were inspired by this venerable article.

Export from source machine – Use list db directory to generate a master script that calls this expdb.cmd for each database:

mkdir %1
cd %1
db2move %1 export > export-%1-db2move-output.txt 2>&1
db2look -d %1 -e -a -l -o db2look-%1.sql > export-%1-db2look-output.txt 2>&1
cd ..

Import into target machine – Use an equivalent master script to call this impdb.cmd:

cd %1
db2 create db %1 using codeset ibm-1252 territory us > import-%1-db2-create-output.txt 2>&1
db2 -tvf db2look-%1.sql > import-%1-db2-output.txt 2>&1
db2move %1 load > import-%1-db2move-output.txt 2>&1
cd ..


We corp-rat developers typically do most real work in some big honkin’ desktop IDE, yet dream of popping into a browser or lightweight shell to get things done. For example, with much of my PHP and Ruby code, I can ssh and vim from anywhere, but my Java work is hopelessly tied to a single laptop. If only we could combine the rich features of an IDE with the shareability and convenience of a cloud environment.

That’s what projects like OrionHub and Codenvy seek to accomplish. Orion seems to have stalled a bit, but since Codenvy is quickly moving forward (and ambitiously taking on Java EE development), I gave it a try.

Codenvy supports a variety of languages, frameworks and cloud platforms, but I only needed Java: for a standalone JAR, and some web code with Spring and Tomcat.

Codenvy works hard to re-create an IDE-style environment in the browser, and does a good job of it. It has an Eclipse-like layout and features, and even many of the same keyboard shortcuts. There were plenty of gaps and differences to remind me that I wasn’t in Kansas anymore, yet I could adjust to those. Even required periodic browser refreshes were OK; they always took me back to some reasonable point.

But it was just too sluggish and buggy for anything but small projects. Granted, Eclipse desktop itself is infamous for stalls, crashes, and sudden disappearances, but this is another level. I suspect some of that can be addressed with faster servers and networks. Indeed, to use this for real work, I’d need the on-premise version where I could control that along with back-end interfaces.

I’ll keep an eye on Codenvy and use it where it fits. It’s great for sharing and collaborating on small bits of code, like a jsFiddle for Java. And maybe soon it’ll be the envy of every developer tied to a desktop IDE.

VoiceObjects by Proxy

During today’s action, I needed to verify new custom HTTP header fields (#HeaderName# properties) in my SOAP web service calls originating from a Voxeo VoiceObjects (VO) call flow.  This meant pointing VO to my favorite trace tool (Fiddler2) running as a proxy on localhost / 8888.  This usually requires setting JVM parameters (proxyHost and proxyPort) on the embedded app server, overriding post URL ports, or using proxycfg, but I suspected VO had an easier way.  Sure enough, after some poking around, I found it:

Repository – Services – (service name) – Proxy ParametersHTTP proxy host and HTTP proxy port

Yet another nice convenience in this market-leading platform.

Careful, They Byte

My son recently updated me on his latest work project, and we chatted about the dangers of treating binary data as characters or words, rather than just streams of bytes.  Nearly everyone must deal with character encoding, but these days byte ordering (endian word) concerns are usually the domain of systems programmers.  Consider:

“A rose is a rose, unless it grows in an Intel garden, in which case it’s an ores or an esor.”


Back in the day, I tossed that around frequently to mock the byte swapping in 16-bit and 32-bit words that we Intel assembly programmers dealt with.

Market Games

Recent record highs have focused a lot of attention on the stock market.  The broad market rise is largely due to Fed actions (quantitative easing and a near zero discount window), creating lots of excess cash and nowhere else good to put it. It’s a risky solution that props up markets while inflation is delayed.

But what about individual stocks? In this rising tide market that can gloss over things, how do you better discern individual winners? Of course, company metrics (fundamentals, earnings, balance sheets, etc.) and movers (news and innovation) are the mainstays. Technical analysis can be helpful, but that tends to focus on surface effects. Can big data look behind the scenes?

Just as tons of consumer market data now drive product marketing decisions, the wealth of available corporate stats increasingly influence stock buy and sell decisions, sometimes to a fault.  In this data mining era, we’re much better at correlation than causation, but that’s often good enough.

The individual investor is perhaps the only truly random walk (or uncertain walk) left in the stock market. Since prices are most influenced by large holders and program trades, movements can be partly predicted by comprehensive mathematical models on the big players and their trading strategies. With enough data and processing power, it’s possible to run rich behavior models in predictive mixed strategy games to forecast prices and actions. There’s been some interesting research in this area, and I think we’ll see more. At least while the current bubble continues to grow.

Entropy Atrophy

derek williams (gatech) has the 2nd best hash.  See the full standings at to the Carnegie Mellon alum for the closest-fit solution to xkcd’s externally-controlled April Fools comic and Skein hash collision contest (always read the alt text).  I was far too hardware-deprived to be nerd-sniped by this one, but there were plenty others who jumped right on it, all motivated by challenge rather than “money.”

That’s encouraging because it seems information theory isn’t taught much anymore (my own alma mater has long since dropped “ICS” for “CS”). Although we now need it most (in our big data and security-starved era), our collective entropy-sophy has atrophied.

In areas like security policy and algorithm design, the cold reality of the pigeonhole principle is too often forgotten.  We often regard hashes as magic, forgetting they’re just bit-twiddling mapping functions and that when the domain is bigger than the range, there will be collisions.  Simple truths like this are ignored in spots ranging from the XBox to ReiserFS.

The winner of xkcd’s contest was still 384 bits shy of a total collision, but every imperfect hash has plenty of clashes.  So be careful out there to win that battle between bit length and processing power, at least while GPUs and quantum computers develop.

BTW, it seems Wikipedia got enough donations from the effort to be good sports about the xkcd-hacking.