Monthly Archives: November 2010

By Proxy

Having misspent years in the trenches of async, LU6.2, Netbios, token ring, and other now-obsolete data comm protocols and interfaces, I’m quite happy that nearly everything now flows over TCP/IP rivers.  Among many other things, this standardization has led to a seemingly endless supply of good tools.

It’s not often that I have to look into the traffic between Point A and Point B, but when I do, I’ve typically used simple tools like PocketSoap’s TcpTrace, pcap-based stuff like ngrep and tcpdump, or sniffers like Wireshark or Solarwinds.  Years ago, I tried Eric Lawrence’s Fiddler, but put it aside because I needed Linux tools.  But I recently needed to debug traffic under Windows, so I gave Fiddler’s successor, Fiddler2, a try.  And I’m glad I did.

Fiddler2 has a dizzying array of features while being very easy to use.  For me, I just needed it for live capture and display of HTTP packets between my client and server.  It doesn’t require manually modifying ports to sit in the middle (it proxies automatically), and it captures data for all sessions.  You can view captured data in every format imaginable, build or modify requests and responses directly from the tool, and set breakpoints and run scripts.  A very nice feature is that it can decrypt HTTPS traffic by using its own cert.  Gone are the days (under Windows, anyway) when I have to switch my HTTPS/SSL services to HTTP for tracing.

It does require configuration steps to use with non-WinINET apps (like PHP), localhost, background services (like IIS), and SSL, but these are straightforward and well-documented.

I’m happy I returned to the Fiddler family, and Fiddler2 will likely remain my tracing proxy of choice for Windows work.  At least until Fiddler3 comes along.

The Friday Fragment

It’s Friday, and time again for the Friday Fragment: our weekly programming-related puzzle.

This Week’s Fragment

Having finished a series of game bots, we’ll have a Thanksgiving change of pace and do another cryptogram.  Crack the following cipher:

OBKR
UOXOGHULBSOLIFBBWFLRVQQPRNGKSSO
TWTQSJQSSEKZZWATJKLUDIAWINFBNYP
VTTMZFPKWGDKZXTJCDIGKUHUAUEKCAR

You can provide the solution or just offer suggestions on how to go about solving it.  For example, I can’t solve it, but based on its background (try some googling), I have ideas about how it might be solved.  To “play along,” post your response as a comment or send it via email.

Last Week’s Fragment – Solution

Last week’s puzzle was to code a simple Reversi competition:

Write code to call competing Reversi bots, flip pieces, and score the results.

Here’s a solution in PHP.  

<?php
  require('reversimove.php);
  $urls[0] = 'http://derekwilliams.us/bots/reversi';
  $urls[1] = $urls[0];  // Your bot here
  $pieces = array('X', 'O');
  $board = str_repeat('.',64);
  $board[27]='O'; $board[28]='X';
  $board[35]='X'; $board[36]='O';
  $player = 0;

  for ($i=0; $i<64; $i++) {;
    $play = file_get_contents($urls[$player].'?board='.$board.'&piece='.$pieces[$player]);
    $board = update_board($board, $play);
    show_board($board);
    $player = $player == 0 ? 1 : 0;
  }

  function show_board($board) {
    $scores = array('.' => 0, 'X' => 0, 'O' => 0);
    echo '<table border="1">';
    for ($i=0; $i<8; $i++) {
      echo '<tr border="1">';
      for ($j=0; $j<8; $j++) {
        $piece = $board[($i*8)+$j];
        $scores[$piece]++;
        echo '<td border="1">'.$piece.'&nbsp;</td>';
      }
      echo '</tr>';
    }
    echo '</table><p>&nbsp;</p>';
    printf("<p>X: %d, O: %d</p>", $scores['X'], $scores['O']);
    if ($scores['.'] == 0)
      exit(printf("<p>%s wins!</p>", $scores['X'] > $scores['O'] ? 'X' : 0));
  }
?>

You can run this solution and other bots from the links at http://derekwilliams.us/bots.  This page also includes full source code, including helper functions.

Microsoft SOAP – Lather, Restart, Repeat

A friend recently asked me to add web service wrappers around some C# .NET code I had written awhile back.  Since Microsoft invented SOAP/XML, it’s not surprising that it’s easy to do in .NET: just create the ASMX file and WebService subclass (with WebService and WebMethod attributes), and .NET handles the rest.  Visual Studio (in my case, VS 2008) even provides templates and wizards to create these for you: nice things like New Web Site – ASP .NET Web Service and Add New Item – Web Service.

I drank the .NET Kool-Aid back when I built this system and constructed with layers of assemblies.  So re-use was fairly straightforward: just add calls to methods (DLL entry points) in the appropriate layer.  Here again, with “Add Reference” and using clauses, Visual Studio makes it very easy.  That is, except for the bugs.

I occasionally got the familiar “type or namespace could not be found” error.  I assumed I had messed up a reference or using clause, or had a problem with the referenced assembly or one of its children.  But every possible cure for these yielded no joy.  Intellisense recognized the references just fine, but the compile/build didn’t.  The solution?  Restart Visual Studio.

This is a recurring bug which seems to move to new areas with each new VS release or fix.  In this case, it seems to be unique to web services referencing .NET assemblies, but it may pop up in other areas.  If I did regular .NET development (like maybe for my day job), I’d keep current on releases and fix levels.  But for this quick exercise, I stuck to the familiar Windoze-land solution: take one restart and call me in the morning.

The Friday Fragment

It’s Friday, and time again for the Friday Fragment: our weekly programming-related puzzle.

This Week’s Fragment

Even T-Rex likes to code games that play themselves, so let’s continue with our Reversi “game bot:”

Write code to call competing Reversi bots, flip pieces, and score the results.  If you’d like, you can build atop the solution we used for tic-tac-toe.

To “play along,” provide either the URL or the code for the solution. You can post it as a comment or send it via email.

Last Week’s Fragment – Solution

Last week’s puzzle was to start a Reversi (a.k.a., Othello) “bot:”

Code a simple REST web service to start playing Reversi.  It should accept a 64-character string representing the current 8×8 “game board” with dot (empty space), X (black), or O (white) in each position (starting from the top left, going left-to-right and down).  It should return a 64-character string with your next move added.   Don’t worry about turning pieces yet, just find the next best move.

Here’s a simple solution in PHP, with room for improvement.  To start playing, pass it ?board=………………………OX……XO………………………&piece=X.

<?php
  if (!ereg('^[XO.]{64}$', $board=$_REQUEST['board']))
    die(str_repeat('X',64));
  if (!ereg('^[XO]$', $mypiece=$_REQUEST['piece']))
    die(str_repeat('.',64));

  if (!is_null($result = find_move($board, $mypiece)))
    exit($result);
  else
    exit($board);

  function find_move($board, $piece) {

    STATIC $plays = array( 0, 7, 56, 63,                  // outer corners
                           2, 5, 16, 23, 40, 47, 58, 61,  // A-squares
                           3, 4, 24, 32, 31, 39, 59, 60,  // B-squares
                          18, 21, 42, 45,                 // inner corners
                          19, 20, 26, 29, 34, 37, 43, 44, // inner sides
                          10, 11, 12, 13, 17, 22, 25, 30, // mid sides
                          33, 38, 41, 46, 50, 51, 52, 53,
                           1, 6, 8, 15, 48, 55, 57, 62,   // C-squares
                           9, 14, 49, 54 );               // X-squares              

    foreach ($plays as $play)
      if ($board[$play] == '.' && can_play($board, $piece, $play)) {
        $board[$play] = $piece;
        return $board;
      }
    return null;
  }
?>

Of course, there are much smarter ways to code Reversi, such as applying common strategies and running look-ahead scenarios, but that’d be more than a fragment.  Years ago, a coworker wrote one for VisualWorks (still in the Cincom Public Repository), and there’s a well-documented implementation at lurgee.net.  If you’re up for coding a smarter version behind the service interface, give it a shot.

DB2 group_concat and other figments

For all that DB2 has to offer, there remain some features found in other relational databases that DB2 still lacks.  Multi-version concurrency control (MVCC) and Oracle-style optimizer hints (between the /*+‘s) often top the list, but there are others.  I’m reminded of these when I find myself recommending something out of habit before I remember DB2 doesn’t have it (rhyme intended).

This happened today with a request for DB2 SQL to build an export command with an explicit column list (no “select *”).  The simple answer was quick:

select ‘export ‘ || group_concat(colname) || ‘ from ‘ || tabname from syscat.columns where … group by tabname

Unfortunately, DB2 doesn’t have group_concat, so I went googling for an alternative.  Some folks offered group_concat, concat_all, and rowtocol implementations as UDFs and stored procedures, while others suggested really ugly recursive SQLs or UNION/CASE alternatives.  None were very general.  One IBMer figured out a way to use DB2’s xmlagg function to hack a solution, which I applied as follows:

select ‘export ‘ || substr(replace(replace(xmlserialize(content xmlagg(xmlelement(name “COLNAME”, colname)) as varchar(1000)), ‘<COLNAME>’, ‘, ‘), ‘</COLNAME>’, ”),3) || ‘ from ‘ || tabname from syscat.columns where … group by tabname

It works, but it’s uuuugly!

I can understand some hesitation to add group_concat to DB2; for example, it’s very easy to exceed column limits.  And architectural differences mean that DB2 doesn’t always need to “clone” everything that other DBs have (there are other ways to do the same thing).  But it’s often helpful to keep up with the database Joneses in one area while you’re passing them in others.

Recent DB2 enhancements (currently committed semantics, literal replacement, optimization profiles, etc.) have shrunk many of these gaps, but many remain to be closed entirely.  With the continued rapid addition of new DB2 features, perhaps we’ll see some of these soon.

Never Don’t Buy Version 4.0 of Anything

As I’ve mentioned before, TOAD is one of my regular tools for database work.  At least it used to be.

I’ve had to keep upgrading TOAD for DB2 to stay current with DB2 releases, but the 4.0 release is so buggy that it’s unusable.  The 4.7 Commercial Version has been out for awhile now, but there’s no sign of those fixes and enhancements coming to the freeware version anytime soon.  Folks have been asking in forums, but there’s no meaningful response.

It’s seems that since the TOAD first donned that Quest T-shirt, this formerly independent tool has become increasingly more constrained.  Ever more menu options are grayed out with the “(Commercial Feature)” taunt.  Simple things that worked in prior releases are broken in the 4.0 freeware versions.  Perhaps they work just fine in the paid commercial version.

I’ve purchased Quest tools in the past, so cost isn’t the issue: availability is.  TOAD is one of those tools that I’ve frequently recommended largely because it was just one free download away.  If freedom and ubiquity weren’t so very important, we might all be coding in CFScript.

Quest is a great company, but TOAD has not done well lately under its ownership (don’t get me started on the lack of a *ix version).  I only wish there was a TOra for DB2.

This happens with some companies that take over a free, independent, or open source project: they take it in under their wing, lock it down, and look for improper ways to capitalize on it.  It can make money in the short run, but let’s be up front about it: state your intentions and either spin off a community version or kill off the freeware version altogether.  This “slippery slope” behavior is right down there with snake oil salesman and Oracle.

The Friday Fragment

It’s Friday, and time again for the Friday Fragment: our weekly programming-related puzzle.

This Week’s Fragment

Let’s do another “game bot” – this time, Othello (a.k.a., Reversi):

Code a simple REST web service to start playing Reversi.  It should accept a 64-character string representing the current 8×8 “game board” with dot (empty space), X (black), or O (white) in each position (starting from the top left, going left-to-right and down).  It should return a 64-character string with your next move added.   Don’t worry about turning pieces yet, just find the next best move.  We’ll work on flipping pieces in next week’s fragment.

To “play along,” provide either the URL or the code for the solution. You can post it as a comment or send it via email.

Last Week’s Fragment – Solution

Last week’s puzzle continued with our a tic-tac-toe “bot”:

Write code to score the results of calling two tic-tac-toe web services playing each other.  Check for wins, ties, and illegal moves.

Here’s a solution with my bot competing against itself (not surprisingly, it ends in a tie).  Of course, it still places a lot of trust in the bots to behave, but we’re keeping things simple here.

<?php
  $urls[0] = 'http://derekwilliams.us/bots/tictactoe';
  $urls[1] = $urls[0];  // Your bot here
  $board = '         ';
  $turn = 'X';

  for ($round=0; $round<5; $round++) {
    for ($i=0; $i<2; $i++) {
      $board = file_get_contents($urls[$i].'?board=|'.rawurlencode($board).'|');
      $board = substr($board, 1, 9);
      show_board($board);
      if (!is_null($result = score_board($board, $round, $turn)))
        exit(show_result($result));
      elseif ($round==4)
        exit(show_result('Tie'));
      $turn = $turn == 'X' ? 'O' : 'X';
    }
  }

  function score_board($board, $round, $turn) {
    STATIC $wins = array( array(0,1,2), array(3,4,5), array(6,7,8),
                          array(0,3,6), array(1,4,7), array(2,5,8),
                          array(0,4,8), array(2,4,6) );

     if (substr_count($board, $turn) != $round + 1)
       return 'Wrong number of '.$turn.'s';
     for ($i=0; $i<8; $i++) {
       for ($j=0, $piececount=0; $j<3; $j++) {
         if ($board[$wins[$i][$j]] == $turn)
           $piececount++;
       }
       if ($piececount == 3)
         return $turn.' wins!';
     }
   }

   function show_board($board) {
     echo '<table border="1">';
     for ($i=0; $i<3; $i++) {
       echo '<tr border="1">';
       for ($j=0; $j<3; $j++)
         echo '<td border="1">'.$board[($i*3)+$j].'&nbsp;</td>';
       echo '</tr>';
     }
     echo '</table><p>&nbsp;</p>';
   }

   function show_result($result) {
     return printf('<p><b>%s</b></p>', $result);
   }
?>

You can run this solution and the bots themselves from the links at http://derekwilliams.us/bots.

Stephen told me of a tic-tac-toe bot he wrote, but, alas, I forgot the full URL.  I’ll try to post it along with next week’s puzzle.

App Inventor

There are some things I try just for the sheer novelty.  As a fan of both Android and visual programming (when done right), I couldn’t resist giving Google’s App Inventor a try.

I confess that I don’t quite understand the “market” for App Inventor; after all, the “regular” Android development environment (the Eclipse ADT plug-in atop the Android SDK) is really nice.  I’m not sure there’s a need to oversimplify and hide things, but perhaps there are a lot non-programmers out there who want to develop Android apps.  Or programmers who don’t like Java.  Or Oraclers who don’t like Harmony.  At any rate, while none of my kids have their own Android smart phones, I suppose it could be a nice learning environment for any youngsters wanting to start with “building blocks” instead of code.

Perhaps the only challenge to Android development setup is getting the ADB USB driver working (in my case, under Windows 7 64-bit): it requires some manual work-arounds.  Since App Inventor uses the same drivers, this might entangle some; for me, that was a price I had already paid back when I installed the Android SDK.  I suspect Google will soon get Microsoft’s blessing to make the Windows automatic driver install work, to benefit both tools.

“Developing” with a web page and a small Java app (the Java-web-started Blocks Editor) feels a bit claustrophobic, but I was pleasantly surprised at how functional this “IDE” is.  Unlike the Eclipse ADT, the App Inventor web tool doesn’t provide virtual devices (AVDs): everything goes directly to my phone as I work, courtesy of the JNLP.  This is smooth and interesting, but was a nuisance when I got a phone call while working.

Still, the Scratch-inspired Blocks Editor is an interesting paradigm, the shapes are really cool, and interlocking them is like assembling a jigsaw puzzle.  I found myself writing meaningless “programs” just to experiment and see how large a mousetrap I could create.  At some (early) point this breaks down, hits limits, or becomes unwieldy, so it’s back to “real” Android development when that happens.  It would be nice if App Inventor and the ADT were somehow integrated; that way, if you run out of visual programming “steam” (and simply must code), you wouldn’t have to start over.

App Inventor is a very nice tool for what it does, so I recommend it to anyone who wants to build a simple Android app, or just wants to dip a toe into the Android development pond.

The Friday Fragment

It’s Friday, and time again for the Friday Fragment: our weekly programming-related puzzle.

This Week’s Fragment

We’ll continue in our tic-tac-toe “game bot” vein with the next step:

Write code to score the results of calling two tic-tac-toe web services playing each other.  Check for wins, ties, and illegal moves.

To “play along,” provide either the URL or the code for the solution. You can post it as a comment or send it via email.  If you’d like, you can build atop results from our last Friday Fragment.

Last Week’s Fragment – Solution

Last week’s puzzle was a tic-tac-toe “bot”:

Code a simple REST web service to play tic-tac-toe.  It should accept a nine-character string representing the current “game board” with space, X, or O in each position (starting from the top left, going left-to-right and down).  It should return a nine-character string with your next move added.

Here’s my Q&D, brute-force solution in PHP, with room for improvement.  I added vertical bar wrappers to preserve spaces on the GET request, so pass it ?board=|         | to start.

<?php

   if (!ereg('^\|{1}[XO ]{9}\|{1}$', $board=$_REQUEST['board']))
     die('|XXXXXXXXX|');
   $board = substr($board, 1, 9);
   $mypiece = (substr_count($board, ' ') & 1) ? 'X' : 'O';

   // The best move is with two plays and one space to win or block.
   // The rest is not much better than random.
   for ($plays=2; $plays>=0; $plays--)
     for ($spaces=1; $spaces<4-$plays; $spaces++)
       if (!is_null($result = find_move($board, $mypiece, $spaces, $plays)))
         exit('|'.$result.'|');

   die('|OOOOOOOOO|');

   function find_move($board, $piece, $spaces, $plays) {

     $wins = array( array(0,1,2), array(3,4,5), array(6,7,8),
                    array(0,3,6), array(1,4,7), array(2,5,8),
                    array(0,4,8), array(2,4,6) );

     for ($i=0; $i<8; $i++) {
       $win=$wins[$i];
       for ($j=0, $mycount=0, $spacecount=0; $j<3; $j++) {
         if ($board[$win[$j]] == $piece)
           $mycount++;
         elseif ($board[$win[$j]] == ' ') {
           $spacecount++;
           $spaceplace = $j;
         }
       }
       $opponentcount = 3 - $mycount - $spacecount;

       if ($spacecount == $spaces) {
         if ($mycount == $plays) {              // win
           $board[$win[$spaceplace]] = $piece;
           return $board;
         } elseif ($opponentcount == $plays) {  // block
           $board[$win[$spaceplace]] = $piece;
           return $board;
         }
       }
     }
     return null;
   }
?>

Can you write a tic-tac-toe bot that’ll beat it?

db2top

I usually don’t have (nor want) direct, real-time access to production databases, yet I’m sometimes asked to diagnose and correct performance issues with them.  So my typical MO is to collect periodic snapshot files (driven by cron or schtasks) and run them “offline” through a set of tools I’ve developed for postmortem analysis.  These tools of mine measure and report trends, calculate deltas, do performance calculations, and flag areas of concern.

But occasionally, I do need to monitor databases in real time.  For this, I’ve relied on third-party tools, such as Quest Spotlight, the free and simple db2mon, db2pd and get snapshot commands, and even Windows perfmon against the DB2 counters.  All the while, I wondered why IBM didn’t provide similar real-time monitoring tools.

Top for DB2

But IBM recently jumped into the game with the db2top utility for Linux and AIX.  As the name implies, it’s like top for DB2.  It uses a text user interface (it’s curses-based) which, despite the retro-80’s feel, is a benefit.  That is, it’s super-fast (jumping between views is quick and easy), and, of course, it runs from a command shell, so no X required.

Db2top shows just about everything available in DB2’s snapshots, and then some.  For example, it automatically calculates things like bufferpool hit ratios, sort overflow percentages, average I/O times, async read percentages, average execution times, and lock chains.  In most views, you can use the arrow keys to scroll right and left to see additional details; I found it useful to increase the screen width to see more without scrolling.  But, of course, the real benefit is watching instantaneous updates and deltas.  This avoids having to “do the math” yourself by processing a series of snapshot files.

With Any More Features, We’d Run Out of Alphabet

There are some nice tie-ins, like the ability to grab an instantaneous snapshot (S) for the corresponding screen, and run explains (L) directly from the Dynamic SQL screen.  Both toss the results directly into vi for viewing.  You can even force applications directly from the sessions (applications) view (bwaaaahahaha!)  And navigation aids like column sorting (Z, z) and screen “freezing” (f) come in very handy when drilling in for closer analysis, particularly when there’s a lot going on.  Quick reference help (h) is available in case you forget these keystrokes.

The bottlenecks (B) screen is useful for finding the “problem children” – those agent(s) consuming the most of each type of DB2 resource: CPU, memory, I/O, locks, sorts, etc.  This makes for a good starting point when jumping into a crisis.

The tool also supports capturing snapshots over time, running analysis reports, exporting to Excel, and replaying results.  I haven’t done much with these features yet, but they may very well take the place of my home-grown snapshot analysis tools.

Just What You Need

Before this, if you told me you had “Spotlight for a command shell,” I’d think you were crazy.  But db2top comes remarkably close.  Overall, it has a lot going for it: ubiquity (it’s there with every DB2 9.x *ix install), speed, and just the monitoring tools I need (no more, no less).  There’s not much lacking, but the following additions would be nice:

  • Extensibility (or source code).  For example, I could then add my own performance calculations – things like log, victim, and threshold trigger ratios for page cleaners.
  • Tie-ins to event monitors, at least db2detaildeadlock.
  • A Windows version, even if it required Cygwin.

Yet I’d prefer it miss a few of these flourishes than be laden with extra features that only add bulk or cost, or that just get in the way.  Too many other tools fall into that trap.