Android back button and MediaController

I know how to take control of the back button. I have a VideoView embedded in a FrameLayout. My question is when the video pops up, the video controls are present for a few seconds. Hitting the back button while they are visible hides the video controls. Is there a way to ignore that function and do the next back action as if the video controls weren't visible?

The reason I ask is if I really do want to go back, I must hit the back button twice; once to hide the controls and second to actually go back