Tag Archives: Web

Just the Facts

Local news sites are notorious for low signal-to-noise ratios.  The news content is good, but that’s often crowded out by excessive ads, Flash videos, runaway JavaScript, and animated GIFs. These things make 90s websites look clean and elegant. I get seasick visiting them.

My typical antidote has been to just stick to RSS, and let blockers like Chrome’s Click to Play squelch things when I have to visit the site.  But the Atlanta Journal-Constitution (AJC) recently broke their RSS feeds while at the same time expanded their JavaScript and floating div monstrosities. What’s a guy to do when he just wants to read the latest Falcons and Georgia Tech news?

Well, I took the nuclear solution and went with lynx.  Yes, lynx: the old text mode browser. Whenever I want to read content from the AJC or similar news sites, I just fire it off from the command line and browse away. It works well, and I can do a quick news check in no time. Hopefully, the AJC won’t start disallowing or punishing lynx use.

jQuery Mobile

This week, I needed a quick mobile (smart phone) proof-of-concept for one of our web app products.  Since speed of development and broad platform support were more important than native device features, I took the mobile web route.  And as a jQuery fan, I chose jQuery Mobile (JQM).

JQM emphasizes doing as much as possible with simple HTML5 and CSS.  It uses data-* custom attributes to know how to add extra behaviors, rendered and tailored as needed for the target device.  Indeed, I found I could do most things with just markup and server-side code, and very little additional JavaScript.

JQM makes AJAX calls by default, but normally expects the results to be new page divs to add to the navigation stack.  This is different from my standard web app, where $.ajax calls return only data (XML and JSON), with bits of JavaScript to update DOM components from the results.  I decided to play along and add div wrappers around results, letting JQM navigate to them as simple linked pages (href=, action=) and dialogs (data-rel=”dialog”).  The end result was cleaner code (and smoother flow than standard postbacks would be), although I did give up a bit in look and feel.

Native app development requires real devices or emulators (such as the Android Virtual Device, AVD), but with a pure mobile web app, I was able to just use Chrome with the nice Ripple extension.  This removed all delays from the code-run cycle: yet another benefit of the mobile web / HTML5 world.

I went with a standard theme and did not (yet) need the extra step of packaging for offline use or adding a native wrapper (like PhoneGap/Cordova).  But those processes appear straightforward when the time comes.  The were a few little glitches along the way, but nothing I couldn’t work around with a little JavaScript.

jQuery Mobile is perhaps the go-to framework for HTML5 mobile web apps.  Certainly for those of us who are sold on the benefits of jQuery.

Secret Handshake

Since security is king in my corp-rat world, standards dictate that my public web services be accessed via mutual authentication SSL.  The extra steps this handshake requires can be tedious: exchanging certs, building keystores, configuring connections, updating encryption JARs, etc.  So when helping developers of a third party app call in, it’s useful to provide a standard tool as a non-proprietary point of reference.

This week I decided to use soapUI to demonstrate calls into my web services over two-way SSL.  The last time I did something like this, I used keytool and openssl to build keystores and convert key formats.  But this go ’round I stumbled across this most excellent post which recommends the user-friendly Portecle tool, and steps through the soapUI setup.

Just a few tips to add:

  • SoapUI’s GUI-accessible logs (soapUI log, http log, SSL Info, etc.) are helpful for diagnosing common problems, but sometimes you have to view content in bin\soapui-errors.log and error.log.   Take a peek there if other diags aren’t helpful.
  • SoapUI doesn’t show full details of the server/client key exchange.  You can get more detailed traces with the simple curl -v or curl –trace; for example:

curl -v -E mykey.pem https://myhost.com/myservice

Happy handshaking!

Friday Fixes

It’s Friday, and time again for some Friday Fixes: selected problems I encountered during the week and their solutions.

Today’s post will be the last of the Friday Fixes series.  I’ve received some great feedback on Friday Fixes content, but they’re a bit of a mixed bag and therefore often too much at once. So we’ll return to our irregularly unscheduled posts, with problems and solutions by topic, as they arrive.  Or, as I get time to blog about them.  Whichever comes last.

More Servlet Filtering

On prior Fridays, I described the hows and whys of tucking away security protections into a servlet filter. By protecting against XSS, CSRF, and similar threats at this lowest level, you ensure nothing gets through while shielding your core application from this burden.  Based on feedback, I thought I’d share a couple more servlet filter security tricks I’ve coded.  If either detects trouble, you can redirect to an error page with an appropriate message, and even kill the session if you want more punishment.

Validate IP Address (Stolen session protection)

At login, grab the remote IP address: session.setAttribute(“REMOTE_IP_ADDR”, request.getRemoteAddr()). Then, in the servlet filter, check against it for each request, like so:

private boolean isStolenSession(HttpServletRequest request) {    
    String uri = request.getRequestURI();
    if (isProtectedURI(uri)) {
        HttpSession session = request.getSession(false);
        String requestIpAddress = request.getRemoteAddr();            
        if (session != null && requestIpAddress != null) {
            String sessionIpAddress = (String) session.getAttribute("REMOTE_IP_ADDR");
            if (sessionIpAddress != null)
                return !requestIpAddress.equals(sessionIpAddress);
        }
    }
    return false;
}

Reject Unsupported Browsers

There may be certain browsers you want to ban entirely, for security or other reasons.  IE 6 (“MSIE 6”) immediately comes to mind.  Here’s a quick test you can use to stop unsupported browsers cold.

private boolean isUnsupportedBrowser(HttpServletRequest request) {
    String uri = request.getRequestURI();
    if (isProtectedURI(uri)) {
        String userAgent = request.getHeader("User-Agent");
        for (String id : this.unsupportedBrowserIds) {
            if (userAgent.contains(id)) {
                return true;
            }
        }
    }
    return false;
}

Community Pools

In my last Friday Fixes post, I described how to use Apache DBCP to tackle DisconnectException timeouts.  But as I mentioned then, if your servlet container is a recent version, it will likely provide its own database connection pools, and you can do without DBCP.

When switching from DBCP (with validationQuery on) to a built-in pool, you’ll want to enable connection validation.  It’s turned off by default and the configuration settings are tucked away, so here’s a quick guide:

Servlet Container Configuration Steps
Tomcat 7 (jdbc-pool) Add the following to Tomcat’s conf/server.xml, in the Resource section containing your jdbc JNDI entry:

  • factory=”org.apache.tomcat.jdbc.pool.DataSourceFactory”
  • testOnBorrow=”true”
  • validationQuery=”select 1 from sysibm.sysdummy1″
WebSphere Set the following in Admin Console:

Resources – JDBC – Data sources > (select data source) > WebSphere Application Server data source properties:

  • Check “Validate new connections”, set “Number of retries” to 5 and set “Retry interval” to 3.
  • Check “Validate existing pooled connections”, and set “Retry interval” to 3.
  • Set “Validation options – Query” to “select 1 from sysibm.sysdummy1”.
WebLogic Set the following in the Administration Console:

JDBC – Data Sources – (select data source) – Connection Pool – Advanced:

  • Check “Test Connections on Reserve”
  • Set “Test Table Name” to “SQL SELECT 1 FROM SYSIBM.SYSDUMMY1”
  • Set “Seconds to Trust an Idle Pool Connection” to 300.

Adjust intervals as needed for your environment.  Also, these validation SQLs are for DB2; use equivalent SQLs for other databases.

Auto Logoff

JavaScript-based auto-logoff timers are a good complement to server-side session limits.  Such are especially nice when coupled with periodic keep-alive messages while the user is still working.  The basic techniques for doing this are now classic, but new styles show up almost daily.  I happen to like Mint‘s approach of adding a red banner and seconds countdown one minute before logging off.

Fortunately, Eric Hynds created a nice jQuery-based Mint-like timer that we non-Intuit folks can use.  It uses Paul Irish’s idleTimer underneath.

Dropping it in is easy; just include the two JS files and add the idletimeout div into your common layout.  I tucked the boilerplate JavaScript into a simple common JSPF, like so:

<%@ taglib uri="http://java.sun.com/jsp/jstl/core" prefix="c"%>
<script src="<c:url value="/script/jquery.idletimer.js"/>"></script>
<script src="<c:url value="/script/jquery.idletimeout.js"/>"></script>
<script>
var idleTimeout = <%= SettingsBean.getIdleTimeout() %>;
if (idleTimeout > 0) {
  $.idleTimeout('#idletimeout', '#idletimeout a', {
	idleAfter: idleTimeout,
	pollingInterval: 30,
	keepAliveURL: '<c:url value="/keepalive" />',
	serverResponseEquals: 'OK',
	onTimeout: function(){
		$(this).slideUp();
		allowUnload();
		window.location = '<c:url value="/logout" />';
	},
	onIdle: function(){
		$(this).slideDown(); // show the warning bar
	},
	onCountdown: function( counter ){
		$(this).find("span").html( counter ); // update the counter
	},
	onResume: function(){
		$(this).slideUp(); // hide the warning bar
	}
  });
}

Inanimate IE

Among Internet Explorer’s quirks is that certain page actions (clicking a button, submitting, etc.) will freeze animated GIFs.  Fortunately, gratuitous live GIFs went out with grunge bands, but they are handy for the occasional “loading” image.

I found myself having to work around this IE death today to keep my spinner moving.  The fix is simple: just set the HTML to the image again, like so:

document.form.submit();
// IE stops GIF animation after submit, so set the image again:
$("#loading").html(
 '<img src="<c:url value="/images/loading.gif"/>" border=0 alt="Loading..." />');

IE will then wake up and remember it has a GIF to attend to.

Dissipation

The trouble with keeping your data in the cloud is that clouds can dissipate.  Such was the case this week with the Nike+ API.

Nike finally unveiled the long-awaited replacement for their fragile Flash-based Nike+ site, but at the same time broke the public API that many of us depend on.  As a result, I had to turn off my Nike+ Stats code and sidebar widget from this blog.  Sites that depend on Nike+ data (dailymileEagerFeet, Nike+PHP, etc.) are also left in a holding pattern.  At this point, it’s not even clear if Nike will let us access our own data; hopefully they won’t attempt a Runner+-like shakedown.

This type of thing is all too common lately, and the broader lesson here is that this cloud world we’re rushing into can cause some big data access and ownership problems.  If and when Nike lets me access my own data, I’ll reinstate my Nike+ stats (either directly, or through a plugin like dailymile’s).  Until then, I’ll be watching for a break in the clouds.

Broken Tiles

I encountered intermittent problems where Apache Tiles 2.2.2 where concurrency issues cause it to throw a NoSuchDefinitionException and render a blank page.  There have been various JIRAs with fixes, but these are in the not-yet-released version 2.2.3.  To get these fixes, update your Maven pom.xml, specify the  2.2.3-SNAPSHOT version for all Tiles components, and add the Apache snapshot repository:

<repository>
    <id>apache.snapshots</id>
    <name>Apache Maven Snapshot Repository</name>
    <url>http://repository.apache.org/snapshots</url>
</repository

Hopefully 2.2.3 will be released soon.

Toolbox Linkapalooza

A few favorite tools I used this week:

  • WebScarab – Nice Ethical Hacking tool.  Among many features, it exposes hidden form fields for editing.
  • Burp Suite – Similar to WebScarab; its proxy and intruder features make packet capture, modification, and replay a snap.
  • Runner’s World Shoe Advisor.

Friday Fixes

It’s Friday, and time again for some Friday Fixes: selected problems I encountered during the week and their solutions.

I got a friendly reminder this morning that I’ve neglected my Friday Fixes postings of late.  I didn’t say they’d be every Friday, did I?  At any rate, here are some things that came up this week.

Tabbed Freedom v. Logoff Security

Tabbed browsing is a wonderful thing, but its features can become security concerns to corp-rat folks who mainly use their browsers for mission critical apps.  For example, with most browsers, closing a tab (but not the browser itself) does not clean up session cookies. Yet those security first guys would like to have a way to trigger a log off (kill the session) on tab close.

This is a common request, but there’s no straightforward solution. As much as I’d like browsers to have a “tab closed” event, there isn’t one.  The best we can do is hook the unload event which is fired, yes, when the tab is closed, but also anytime you leave the page: whether it’s navigating a link, submitting a form, or simply refreshing.  So the trick is to detect and allow acceptable unloads.  Following is some JavaScript I pieced together (into a common JSPF) based loosely on various recommendations on the web.

  var isOkToUnload = false;
  var logoffOnClose = '<c:out value="${settingsBean.logoffOnClose}" />';
 
  function allowUnload() {
     isOkToUnload = true;
  }
  function monitorClose() {
     window.onbeforeunload = function() {
        if (!isOkToUnload)
           return "If you leave this page, you will be logged off.";
     }
     window.onunload = function() {
        if (!isOkToUnload) {
           $.ajax({
              async: false, type: "POST",
                   url: '<c:url value="/basiclogout"/>' });
        }
     }   
     // Add other events here as needed
     $("a").click(allowUnload);      
     $("input[type=button]").click(allowUnload);      
     $("form").submit(allowUnload);                     
  }
 
  $(document).ready(function() {
     if (logoffOnClose === 'Y')
        monitorClose();
  });

This triggers on refresh, but that’s often a good thing since the user could lose work; Gmail and Google Docs do the same thing when you’re editing a draft.  It’s a good idea to make this behavior configurable, since many folks prefer the freedom of tabbed browsing over the security of forcing logoff.

DBCP Has Timed Out

Right after mastering the linked list, it seems every programmer wants to build a database connection pool.  I’ve built a couple myself, but this proliferation gets in the way of having a single golden solution that folks could rally around and expect to be supported forever.

Such was the story behind Apache DBCP: it was created to unify JDBC connection pools.  Although it’s broadly used, it’s over-engineered, messy, and limited. So it, too, fell by the wayside of open source neglect.  And since nearly all servlet containers now provide built-in connection pools, there’s really no use for DBCP anymore.

Yet I found myself having to fix DisconnectException timeouts with an existing DBCP implementation, typically stemming from errors like:  A communication error has been detected… Location where the error was detected: Reply.fill().

After trying several recovery options, I found that DBCP’s validationQuery prevented these, at the cost of a little extra overhead.  Although validationQuery can be configured, I didn’t want additional setup steps that varied by server.  So I just added it to the code:

  BasicDataSource ds = new BasicDataSource();
  // ...
  ds.setValidationQuery("select 1 from sysibm.sysdummy1");
  ds.setTestOnBorrow(true)

In the next pass, I’ll yank out DBCP altogether and configure pools in WebSphere, WebLogic, and Tomcat 7.  But this gave me a quick fix to keep going on the same platform.

Aggregation Dictation

Weird: I got three questions about aggregating in SQL on the same day.  Two of them involved OLAP syntax that’s somewhat obscure, but quite useful.  So if you find yourself with complications from aggregation aggravation, try one of these:

  • Doing a group by and need rollup totals across multiple groups?  Try grouping sets, rollup, and cube.  I’ve written about these before; for example, see this post.
  • Need to limit the size of a result set and assign rankings to the results?  Fetch first X rows only works fine for the former, but not the latter.  So try the ranking and windowing functions, such as row_number, rank, dense_rank, and partition by.  For example, to find the three most senior employees in each department (allowing for ties), do this:

      SELECT * FROM
       (SELECT rank() OVER(partition BY workdept ORDER BY hiredate ASC) 
        AS rank, firstnme, lastname, workdept, hiredate 
        FROM emp) emps
      WHERE rank <= 3
      ORDER BY workdept

    Co-worker Wayne did a clever row_number/partition by implementation and added a nice view to clarify its meaning.

Linkapalooza

Some interesting links that surfaced (or resurfaced) this week:

Friday Fixes

It’s Friday, and time again for some Friday Fixes: selected problems I encountered during the week and their solutions.

This week’s challenges ran the gamut, but there’s probably not much broad interest in consolidated posting for store-level no advice chargebacks, image format and compression conversion, SQLs with decode(), 798 NACHA addenda, or many of the other crazy things that came up.  So I’ll stick to the web security vein with a CSRF detector I built.

Sea Surf

If other protections (like XSS) are in place, meaningful Cross-Site Request Forgery (CSRF) attacks are hard to pull off.  But that usually doesn’t stop the black hats from trying, or the white hats from insisting you specifically address it.

The basic approach to preventing CSRF (“sea surf”) is to insert a synchronizer token on generated pages and compare it to a session-stored value on subsequent incoming requests.  There are some pre-packaged CSRF protectors available, but many are incomplete while others are bloated or fragile.  I wanted CSRF detection that was:

  • Straightforward – Please, no convoluted frameworks nor tons of abstracted code
  • Complete – Must cover AJAX calls as well as form submits, and must vary the token by rendered page
  • Flexible – Must not assume a sequence of posts or limit the number of AJAX calls from one page
  • Unobtrusive – Must “drop in” easily without requiring @includes inside forms.

I also wanted to include double submit protection, without having to add another filter (certainly no PRG filters – POSTs must be POSTs).  Here below is the gist of it.

First, we need to insert a token.  I could leverage the fact that nearly all of our JSPs already included a common JSPF file, so I just added to that.  The @include wasn’t always inside a form so I added the hidden input field via JavaScript (setToken).  I used a bean to keep the JSPF as slim as possible.

<jsp:useBean id="tokenUtil" class="com.writestreams.TokenUtil">
	<% tokenUtil.initialize(request); %>
</jsp:useBean>
 
<script>
	var tokenName = '<c:out value="${tokenUtil.tokenName}" />';
	var tokenValue = '<c:out value="${tokenUtil.tokenValue}" />';	
 
	function setToken(form) {
		$('<input>').attr({
   			type: 'hidden',
    		id: tokenName,
    		name: tokenName,
    		value: tokenValue
		}).appendTo(form);	
	}
 
	$("body").bind("ajaxSend", function(elm, xhr, s){
		if (s.type == "POST") {
			xhr.setRequestHeader(tokenName, tokenValue);
   		}
	});	
</script>

I didn’t want to modify all those $.ajax calls to pass the token, so the ajaxSend handler does that.  The token arrives from AJAX calls in the request header, and from form submits as a request value (from the hidden input field); that gives the benefit of being able to distinquish them.  You could use a separate token for each if you’d like.

The TokenUtil bean is simple, just providing the link to the CSRFDetector.

public class TokenUtil {
	private String tokenValue = null;
	public void initialize(HttpServletRequest request) {
		this.tokenValue = CSRFDetector.createNewToken(request);
	}	
	public String getTokenValue() {
		return tokenValue;
	}
	public String getTokenName() {
		return CSRFDetector.getTokenAttribName();
	}	
}

CSRFDetector.createNewToken generates a new random token for each page render and adds it to a list stored in the session.  It does JavaScript-encoding in case there are special characters.

public static String createNewToken(HttpServletRequest request) {
	HttpSession session = request.getSession(false);		
	String token = UUID.randomUUID().toString();		
	addToken(session, token);
	return StringEscapeUtils.escapeJavaScript(token)	
}

A servlet filter (doFilter) calls CSRFDetector to validate incoming requests and return a simple error string if invalid.  You can limit this to only validating POSTs with parameters, or extend it to other requests as needed.  The validation goes like this:

public String validateRequestToken(HttpServletRequest request) {			
	HttpSession session = request.getSession(false);
	if (session == null) 
		return null;	// No session established, token not required
 
	String ajaxToken = decodeToken(request.getHeader(getTokenAttribName()));
	String formToken = decodeToken(request.getParameter(getTokenAttribName()));
	if (ajaxToken == null && formToken == null) 		
		return "Missing token";
 
	ArrayList<String> activeTokens = getActiveTokens(session);
	if (ajaxToken != null) {
		// AJAX call - require the token, but don't remove it from the list
		// (allow multiple AJAX calls from the same page with the same token).
		if (!activeTokens.contains(ajaxToken))		
			return "Invalid AJAX token";
	} else {
		// Submitted form - require the token and remove it from the list
		// to prevent any double submits (refresh browser and re-post).
		if (!activeTokens.contains(formToken)) 			
			return "Invalid form token";				
		activeTokens.remove(formToken);
		setActiveTokens(session, activeTokens);
	}
	return null;		
}

There you have it.  There are several good tools available for testing; I recommend OWASP’s CSRFTester.

Linkapalooza

Some useful / interesting links that came up this week:

  • Grep Console – Eclipse console text coloring (but no filtering)
  • Springpad – Think Evernote++
  • Pocket – New name for ReadItLater
  • RFC 2616 – The HTTP 1.1 spec at, well, HTTP

Authenticate This

I had a good lunch today with a friend who wanted to quickly set up simple (yet strong) authentication on a Tomcat web server using his own login page.  Since forms authentication is built into all J2EE web servers (and ASP .NET servers, for that matter), it’s quite easy.

In summary, the steps for Tomcat are:

  1. Add security-constraints to WEB-INF/web.xml to specify protected resources / folders.   Also include auth-constraints and security-roles for access.
  2. Create user and user role tables and configure the JDBC realm in server.xml.  Or, simply start with defining users directly in tomcat-users.xml; you can always add the database later.
  3. Create the login and login error JSPs, pointing to them from the login-config section of web.xml.  Remember to include the required form element names in the JSP (j_security_check, j_username, j_password, etc.).  Also note that these pages can’t use style sheets and other external files, so you have to (redundantly) embed style information directly into the JSP.

By default, all traffic (including login passwords) isn’t encrypted, so this should only be used with SSL/TLS encryption in place.  That means installing a digital certificate, which is also fairly easy.  That is:

  1. Purchase an SSL certificate.  For initial testing, you can create a self-signed cert using keytool, included with JSSE.
  2. Edit server.xml and enable the SSL (port 8443) connector.  The commands are already present, just un-comment them.
  3. If the server is local, re-start Tomcat, open your browser, and access your site using the https://localhost:8443 URL.  Look for the browser cues for a secure site: padlock icon, green or yellow address bar, etc.

You may eventually switch to more sophisticated methods, like integrating with external security systems for single sign on (e.g., using SAML).  But the simple steps above will get you going quickly with basic, unbreakable authentication.