Detecting Chess moves from successive image differences using OpenCV tools

Hey, I am coding up a simple chess playing robot's vision system, I am trying to improve on some previous research to allow camera and a standard chess set be used and both be allowed to move during the game. So far I can locate the board in an image acquired via web-cam, and I want to detect moves by taking difference of successive images to determine what has changed then use previous information about the board occupancy to detect moves.

My problem is that I can't seem to reliably detect changes at the moment, my current pipeline goes like this: Subtract two images -> Histogram equalize the difference image -> erode and dilate diff image to remove minor changes -> make a binary copy and do distance transform -> Get the largest blob(corresponding to the highest value after DT and flood fill that blob) -> repeat again until DT returns a value small enough to ignore change.

I am coding all this in OpenCV and C++. but my flood fill seem to always either not fill the blobs, hence most cases I just get one change detected. I have tried also using cv::inpaint but that didn't help either. So my question is; am I just using the wrong approach or somehow turing can make the change detection more reliable. In case of the former, could people suggest alternative routes, preferable codable in C++/Python and/or OpenCV in a reasonable time?

thanks