I've stumbled upon the same issue multiple times and I always solved it by adjusting UV coordinates to be at the exact center of four texels you need to fetch (the shared corner of A,B,C and D in your example). For instance:
float2 gatherUV = (floor(uv * textureSize - 0.5) + 1.0) / textureSize;
This approach does not rely on any internal texture filtering details.

I also stumbled accross this. Also if you uses non power of two textures a tiny additional seemingly random non uniform offset gets added. Kind of a pain.

Might be a rounding issue - do you divide by textureSize anywhere?