Onhashchange with browser buttons only

I've got this issue (I'm using jQuery but I'm not restricted to it):

I'm using a combo of Anchor navigation (#id) and Ajax requests. To get the pages to move into place (using anchor navigation) or to fetch information (using Ajax), I use the onhashchange event.

EDIT: I had a little typo. I forgot to check if the mouseDown flag was true and the hashchange event was triggered so I added that if statement.

with jQuery it looks like this: (of course this code is wrapped in a function and initialized on DOM load but it doesn't matter for the question)

$(window).bind('hashchange', function(e) { }

To ensure only browsers supporting the onhashchange reads the code I encapsulate it like this:

if ('onhashchange' in window) {
  $(window).bind('hashchange', function(e) { }
}

My web app is made in such way that I only want the onhashchange event to trigger when I hit the back/forward buttons in the browser. To do that I do like this:

if ('onhashchange' in window) {
  $(window).bind('mousedown hashchange', function(e) { }
}

Now if I click within the viewport I will trigger the mousedown event. If the mousedown event is triggered I know that I didn't click the browser back/forward buttons and I can stop the onhashchange event using a flag like this:

var mouseDown = false;

if ('onhashchange' in window) {
  $(window).bind('mousedown hashchange', function(e) {
    if (e.type === 'mousedown') {
      mouseDown = true;
    }

    if (mouseDown && e.type === 'hashchange') {
      // if the mousedown event was triggered and when the haschange event triggers,
      // we need to stop the hashchange event and restore the mouseDown flag
      mouseDown = false;
      e.stopPropagation();          
    }

    if (!mouseDown && e.type === 'hashchange') {
      // Do the onhashchange stuff here
    }
  }
}

This causes a problem for IE since it seams you cannot bind mouse events to the window object (?). IE will never "see" the mousedown event.

To solve this IE issue I can take the "clientY" property. This property is passed in all event calls in IE and tells you the coordinates of the mouse. If e.clientY is less then 0, the mouse is outside the viewport and I will know that I triggered the onhashchange by clicking the browser back/forward buttons. It now looks like this:

var mouseDown = false;

if ('onhashchange' in window) {
  $(window).bind('mousedown hashchange', function(e) {
    // IE: Use e.clientY to check if the mouse position was within the viewport (i.e. not a nagative value for Y)
    // !IE: Use e.type
    if (e.type === 'mousedown' || e.clientY > 0 ) {
      mouseDown = true;
    }

    if (mouseDown && e.type === 'hashchange') {
      // if the mousedown event was triggered and when the haschange event triggers,
      // we need to stop the hashchange event and restore the mouseDown flag
      mouseDown = false;
      e.stopPropagation();          
    } 
    if (!mouseDown && e.type === 'hashchange') {
      // Do the onhashchange stuff here
    }
  }
}

This solution was working like a charm until I had to add support for navigating with the arrows on the keyboard. Now it doesn't matter where on the screen the mouse is. As long as the IE window is "active", the keydown event listening for keyboard input triggers when hitting the keyboard. This means that the clientY check does not work anymore as intended.

The Problem:

As far as I know, the onhashchange must be bound to the window object. All events must be processed within the same callback function if I want to be able to control one event by listening for another.

How can I get this to work?