How to use std::ifstream to read in a binary file with a wide string path

I am reading a binary file as:

const size_t stBuffer = 256;
char buffer[stBuffer];
std::wstring wPath(L"blah");
std::wifstream ifs(wPath.c_str(), std::wifstream::in | std::wifstream::binary)
while (ifs.good())
{
  ifs.read(buffer, sizeof(buffer));
  ...
}

But I am realizing this is not a true binary read. The ifstream actually reads a byte and converts it to a wide char. So if the binary file has the content 0x112233...ff, I actually read 0x110022003300...ff00.

This doesn't make much sense to me: first, I only need to use a wide fstream because the file name is non Latin. Second, if I say the fstream is binary, why does read read wide chars? The code below does what I want. Is there a way to achieve that using std fstreams?

FILE* ifs = _wfopen(L"blah", L"rb");
while (!feof(ifs))
{
  size_t numBytesRead = fread(buffer, 1, sizeof(buffer), ifs);
  ...
}