Yet another tiny and simple program to visualise the randomness of

binary files written in Java.

http://www.freecx.co.uk/crypto/crypt...andomFile.java

Essentially it just define a new image based on the file size or a given

byte offset - similar to the previously posted Python program - and read

in the given file byte per byte, place a pixel based on the integer of

the byte value at a sequential position. The calculated image will then

be resized to 512 x 512 pixel and this resulting image will be saved at

the same location as the binary file.

Surprisingly this simple routine reveal even more precise if the PRNG

which generated the random binary file is biased.

For example if we again check the binary file

http://www.freecx.co.uk/crypto/crypt...badRNG_4MB.bin

the resulting image show a clearly visible pattern that indicate a

massive bias

http://www.freecx.co.uk/crypto/crypt...MB.bin_rnd.jpg

The same holds for the following image which reveal a massive pattern as well

http://www.freecx.co.uk/crypto/crypt...MB.bin_rnd.jpg

which ist based on the binary file

http://www.freecx.co.uk/crypto/crypt...lyprng_4MB.bin

and of course the extreme pattern in

http://www.freecx.co.uk/crypto/crypt...alline_rnd.jpg

based on the binary file

http://www.freecx.co.uk/crypto/crypt...in.crystalline

All of these binary files fail other tests for randomness, like

Rabbit/Alphabit or the Qualcomm bias test.

Now I would like to highlight how the test behave on checking a JPEG, a PDF and a text file.

First let's look at the result of the JPEG file

http://www.freecx.co.uk/crypto/crypt...262144_rnd.jpg

based on this image

http://www.freecx.co.uk/crypto/crypt..._Chevalier.jpg

Surprisingly it looks quite random even if we would expect to see some pattern.

A test with ENT give us this result

Only the Chi square indicate that the binary is not as random as expectedCode:*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* Entropy = 7.978943 bits per byte. Optimum compression would reduce the size of this 313196 byte file by 0 percent. Chi square distribution for 313196 samples is 8933.78, and randomly would exceed this value less than 0.01 percent of the times. Arithmetic mean value of data bytes is 128.1416 (127.5 = random). Monte Carlo value for Pi is 3.118297285 (error 0.74 percent). Serial correlation coefficient is 0.017011 (totally uncorrelated = 0.0).

in comparison with the greyscale image.

The Rabbit/Alphabit test however reveal that the binary is far from

being random.

Because of the fact that the visualisation tool just collect byteCode:============================================== 313196 byte for testing ======== Running Rabbit Test ========= ========= Summary results of Rabbit ========= Version: TestU01 1.2.3 File: 2D-Example-Binary-Files/Krak_des_Chevalier.jpg Number of bits: 313184 Number of statistics: 38 Total CPU time: 00:00:00.63 The following tests gave p-values outside [0.001, 0.9990]: (eps means a value < 1.0e-300): (eps1 means a value < 1.0e-15): Test p-value ---------------------------------------------- 1 MultinomialBitsOver eps 2 ClosePairsBitMatch, t = 2 2.6e-71 3 ClosePairsBitMatch, t = 4 1.4e-149 4 AppearanceSpacings 1 - eps1 6 LempelZiv 1 - eps1 8 Fourier3 4.1e-11 10 PeriodsInStrings 9.1e-6 11 HammingWeight eps 12 HammingCorr, L = 32 eps 13 HammingCorr, L = 64 eps 14 HammingCorr, L = 128 eps 15 HammingIndep, L = 16 7.4e-6 18 AutoCor 1 - 1.8e-5 20 Run of bits eps 24 RandomWalk1 H 3.9e-8 24 RandomWalk1 M 2.5e-6 25 RandomWalk1 H (L = 1024) 2.6e-8 26 RandomWalk1 H (L = 10016) 4.2e-7 ---------------------------------------------- All other tests were passed *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* ======== Running Alphabit Test ======== ========= Summary results of Alphabit ========= Version: TestU01 1.2.3 File: 2D-Example-Binary-Files/Krak_des_Chevalier.jpg Number of bits: 313184 Number of statistics: 17 Total CPU time: 00:00:00.03 The following tests gave p-values outside [0.001, 0.9990]: (eps means a value < 1.0e-300): (eps1 means a value < 1.0e-15): Test p-value ---------------------------------------------- 1 MultinomialBitsOver, L = 2 eps 2 MultinomialBitsOver, L = 4 eps 3 MultinomialBitsOver, L = 8 eps 4 MultinomialBitsOver, L = 16 eps 5 HammingIndep, L = 16 7.4e-6 7 HammingCorr, L = 32 eps 8 RandomWalk1 H (L = 64) 8.4e-14 8 RandomWalk1 M (L = 64) 1.3e-7 8 RandomWalk1 J (L = 64) 7.9e-6 9 RandomWalk1 H (L = 320) 5.0e-4 9 RandomWalk1 M (L = 320) 2.5e-4 ---------------------------------------------- All other tests were passed

sequentially as they appear in the binary file without any further

calculation for potential bias we can not expect it to be a one-and-only

tool to rely on in regards of finding bias.

A JPEG file has somehow a random structure that the simple visualisation

function displays. The ENT test explains this if we look at the

Arithmetic mean, the Monte Carlo value for Pi and the Serial correlation

coefficient. So we can't expect this visualisation tool to find the far

more detailed results like the Rabbit/Alphabit test.

So let's take a look at two more structured files and check them with the

randomness visualisation function.

First we visualise a PDF file like this one

http://www.freecx.co.uk/crypto/crypt...visited%20.pdf

(which I have downloaded from here http://fse2011.mat.dtu.dk/slides/Att...visited%20.pdf)

The randomoness image clearly show that this is a non-random file

http://www.freecx.co.uk/crypto/crypt...048576_rnd.jpg

Secondly we check this huge text file

http://www.freecx.co.uk/crypto/crypt...-Files/big.txt

and again the resulting image indicate that it is a non-random file

http://www.freecx.co.uk/crypto/crypt...308416_rnd.jpg

As a conclusion I like to say that even if the randomness visualisation

tool is that extremely simple, still it could be part of a test series

because it is able to indicate a binary file generated by a biased PRNG,

as the three examples at in the beginning of this post explain.

All binary files and resulting test images can be found over here

http://www.freecx.co.uk/crypto/crypt...-Binary-Files/

Cheers,

Karl-Uwe

Tweet This+ 1 thisPost To Linkedin