Linux and Windows write timing difference

Dev Shed Forums Sponsor:

November 19th, 2012, 03:47 PM

chipbu

Registered User

Join Date: Oct 2012

Posts: 2

Time spent in forums: 8 m 52 sec
Reputation Power: 0

Linux and Windows write timing difference

Hi guys,

This is not related to syntax or runtime problem. What I am going ask is more about how Linux and Windows handle writing data from buffer to a file. I have this code here, wrapped around a timing block, to write a buffer to a file.

Code:

StartCounter();
   if(rows != fwrite(image, cols, rows, fp)){
      fprintf(stderr, "Error writing the image data in write_pgm_image().\n");
      if(fp != stdout) fclose(fp);
      return(0);
   }

   test = GetCounter();

Code:

#include <shrUtils.h>

#ifdef _WIN32
double PCFreq = 0.0;
__int64 CounterStart = 0;
#endif

#ifdef __linux__
struct timeval ts_start,ts_end;
#endif

void StartCounter()
{
#ifdef _WIN32
	LARGE_INTEGER li;
    if(QueryPerformanceFrequency(&li) == 0)
		printf("QueryPerformanceFrequency failed!\n");

    PCFreq = (float)((li.QuadPart)/1000.0);

    QueryPerformanceCounter(&li);
    CounterStart = li.QuadPart;
#endif

#ifdef __linux__
	gettimeofday(&ts_start, NULL);
#endif
}
double GetCounter()
{
#ifdef _WIN32
	LARGE_INTEGER li;
    QueryPerformanceCounter(&li);
    return (float)((li.QuadPart-CounterStart)/PCFreq);
#endif
	
#ifdef __linux__
	gettimeofday(&ts_end, NULL);
	//time = timespec_sub(ts_end, ts_start);
	return (float)((ts_end.tv_sec - ts_start.tv_sec + 1e-6 * (ts_end.tv_usec - ts_start.tv_usec))*1000.0);
#endif
}

When I measure time to write data, I found out that:
1 - in Linux, the time to write data is linearly proportional to the data size.
2 - in Windows, the time to write data is quadratically proportional to the data size.

I think the fwrite function writes the data line by line to the file, therefore the linear relationship in Linux. But seems like Windows behaves differently. Do you think of any explanation for this?

Any help is greatly appreciate

November 20th, 2012, 04:54 AM

mitakeet

I'm Baaaaaaack!

Dev Shed God 1st Plane (5500 - 5999 posts)

Join Date: Jul 2003

Location: Maryland

Posts: 5,538

Time spent in forums: 2 Weeks 4 Days 2 h 38 m 46 sec
Reputation Power: 242

Measuring performance is tricky and more art than science so the differences you are measuring might be artifacts of the way you designed the measurement rather than anything underlying the OS. Also, the seemingly simple act of trying to measure time is actually a massive problem, most particularly on multi-core/CPU systems, and one that makes measuring small events a nightmare. You don't say if you are measuring with the exact same hardware and if not, then your results are largely meaningless.

As an FYI, fwrite() is implemented using underlying OS-specific API calls and different implementations of the library interface behave differently on different OSs. Meaning, Visual Studio should be your IDE of choice when attempting to optimize on Windows; the use of gcc on Windows, while it _may_ give you high performance, is a more generic compiler. Of course, gcc on Windows has the added complexity of cygwin, though I believe there is a supported port of gcc that works direct on Windows.

How the data is buffered is highly dependent on hardware (i.e., disk) settings and different OSs can 'tweak' drive settings in firmware as they boot, further confounding the issue even when running on identical hardware. There are several layers between your C code call of fwrite() and the actually placement of bytes on the disk (presuming you aren't using a SSD; you are aware of details at that level, I hope) and most of those layers are totally outside your control. It is also possible to tweak OS settings during install and even boot time to modify the way data is cached. Without taking the time to understand all these nitty gritty little details your attempt at measuring write performance is pretty much a waste of time. You could, though, use this as an incentive to learn those types of details, possibly becoming a superlative programmer as a consequence, or you could just choose to ignore them and not worry about things that are difficult (and sometimes impossible) to change. Though I am aware of these issues, I choose to ignore them except at the highest level (meaning if I am doing lots of writes I will get a RAID SSD for maximal throughput, then ignore). Awareness can make you a better programmer when performance is important, but generally speaking, these sorts of issues are almost never an issue in real-world programming.

BTW, I have a writeup on performance programming you can access via my sig if you are interested. It might provide some insights for you.

__________________

My blog, The Fount of Useless Information http://sol-biotech.com/wordpress/
Free code: http://sol-biotech.com/code/.
Secure Programming: http://sol-biotech.com/code/SecProgFAQ.html.
Performance Programming: http://sol-biotech.com/code/PerformanceProgramming.html.
LinkedIn Profile: http://www.linkedin.com/in/keithoxenrider

It is not that old programmers are any smarter or code better, it is just that they have made the same stupid mistake so many times that it is second nature to fix it.
--Me, I just made it up

The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man.
--George Bernard Shaw

November 20th, 2012, 12:27 PM

Scorpions4ever

Banned ;)

Dev Shed God 9th Plane (9000 - 9499 posts)

Join Date: Nov 2001

Location: Woodland Hills, Los Angeles County, California, USA

Posts: 9,385

Time spent in forums: 1 Month 4 Weeks 1 Day 21 h 30 m 25 sec
Reputation Power: 4080

Quote:

Originally Posted by chipbu

I think the fwrite function writes the data line by line to the file, therefore the linear relationship in Linux. But seems like Windows behaves differently. Do you think of any explanation for this?

This is because of the I/O buffering done by the OS. In Linux, you could use setvbuf() to change the file handle from line buffering to full buffering (or unbuffered, if you want it that way).

__________________
Up the Irons
What Would Jimi Do? Smash amps. Burn guitar. Take the groupies home.
"Death Before Dishonour, my Friends!!" - Bruce D ickinson, Iron Maiden Aug 20, 2005 @ OzzFest
Down with Sharon Osbourne