Entity Framework Caching with the Repository Pattern

If I want to implement caching when I am using the repository pattern and the Entity Framework, couldn't I just do some simple logic outside of the Entity Framework to handle the caching?

E.g.

if(Cache[ProductsKey] != null)
{
    return ConvertToProducts(Cache[ProductsKey]);
}
else
{
    var products = repository.Products;
    Cache[ProductsKey] =  products;
    return products;
}

It seems like a lot of people are over-complicating this. Or is doing it this way going to be limiting in some way?