HTTP Response Routing - finding the client IP

I am asking this as I need to retrieve the actual IP address of the machine making an HTTP request (a web service as it happens).

The HTTP Header("REMOTE_ADDRESS") is one place I can get it but there is a problem:

When going through a HTTP caching server the IP address is altered. Apparently the 'X-Forwarded-For' is often where the cache puts the original IP address, but this seems to be at the discretion of the cache itself.

I am not worried about NAT networks, although that is of course a problem too.

Somehow the web server must know where to send the response. Does anyone out there know how one can find this information programatically (.NET preferred).

cheers