Monthly Archives: April 2013

Market Games

Recent record highs have focused a lot of attention on the stock market.  The broad market rise is largely due to Fed actions (quantitative easing and a near zero discount window), creating lots of excess cash and nowhere else good to put it. It’s a risky solution that props up markets while inflation is delayed.

But what about individual stocks? In this rising tide market that can gloss over things, how do you better discern individual winners? Of course, company metrics (fundamentals, earnings, balance sheets, etc.) and movers (news and innovation) are the mainstays. Technical analysis can be helpful, but that tends to focus on surface effects. Can big data look behind the scenes?

Just as tons of consumer market data now drive product marketing decisions, the wealth of available corporate stats increasingly influence stock buy and sell decisions, sometimes to a fault.  In this data mining era, we’re much better at correlation than causation, but that’s often good enough.

The individual investor is perhaps the only truly random walk (or uncertain walk) left in the stock market. Since prices are most influenced by large holders and program trades, movements can be partly predicted by comprehensive mathematical models on the big players and their trading strategies. With enough data and processing power, it’s possible to run rich behavior models in predictive mixed strategy games to forecast prices and actions. There’s been some interesting research in this area, and I think we’ll see more. At least while the current bubble continues to grow.

Entropy Atrophy

derek williams (gatech) has the 2nd best hash.  See the full standings at http://almamater.xkcd.com/best.csvKudos to the Carnegie Mellon alum for the closest-fit solution to xkcd’s externally-controlled April Fools comic and Skein hash collision contest (always read the alt text).  I was far too hardware-deprived to be nerd-sniped by this one, but there were plenty others who jumped right on it, all motivated by challenge rather than “money.”

That’s encouraging because it seems information theory isn’t taught much anymore (my own alma mater has long since dropped “ICS” for “CS”). Although we now need it most (in our big data and security-starved era), our collective entropy-sophy has atrophied.

In areas like security policy and algorithm design, the cold reality of the pigeonhole principle is too often forgotten.  We often regard hashes as magic, forgetting they’re just bit-twiddling mapping functions and that when the domain is bigger than the range, there will be collisions.  Simple truths like this are ignored in spots ranging from the XBox to ReiserFS.

The winner of xkcd’s contest was still 384 bits shy of a total collision, but every imperfect hash has plenty of clashes.  So be careful out there to win that battle between bit length and processing power, at least while GPUs and quantum computers develop.

BTW, it seems Wikipedia got enough donations from the effort to be good sports about the xkcd-hacking.