BinaryFormatter.Deserialize “unable to find assembly” after ILMerge

I have a C# solution with a referenced dll (also C# with the same .Net version). When I build the solution and run the resulting exe, without merging the exe and the referenced dll, everything works fine.

Now I want to merge these into one exe. I run ILMerge and everything appears to work ok. I try to execute the exe and it seems to run just fine until it tries to deserialize an object defined in the referenced dll.

using (Stream fstream = new FileStream(file_path, FileMode.Open))
{
    BinaryFormatter bf = new BinaryFormatter();
    return bf.Deserialize(fstream) as ControlledRuleCollection; 
    // throws unable to find assembly exception
}

Is there maybe some ILMerge option I'm missing here?