#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2011
    Posts
    2
    Rep Power
    0

    Where to find spoken languages statistics?


    Hi,

    I'm doing a hobby project, decrypting ciphers. One powerful method is to use frequency analysis of letters, letter-combinations and words.

    There is a lot information to find on the internet about the English language, but other languages seems harder to find (other than single letter frequencies).

    I guess there must be research done in this area, but I haven't been able to find any. Writing your own algorithm, extracting the information, isn't a hard task, but finding the right composition of texts to analyze probably requires a language specialist.

    Does anyone know if there is any open publications or other resources on this subject?

    / J
  2. #2
  3. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2010
    Posts
    95
    Rep Power
    5
  4. #3
  5. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2011
    Posts
    2
    Rep Power
    0
    Thanks!
    Too bad there aren't more languages listed..
  6. #4
  7. Banned ;)
    Devshed Supreme Being (6500+ posts)

    Join Date
    Nov 2001
    Location
    Woodland Hills, Los Angeles County, California, USA
    Posts
    9,643
    Rep Power
    4248
    You know what, you ought to read the original document that talks about using frequency analysis to crack ciphers. It was written by an ancient Iraqi chap named Al-Kindi. In his very illuminating document, he says (translating it to modern equivalent from memory here):

    "First, find a book that is in the same language of the cipher and count the frequency of the letters from a few pages. Then count the frequency of the letters in the cipher text....".

    So, there's your solution in the very first sentence of his paper. You could probably go quite far by getting your hands on a novel, online newspaper, magazine etc. in another language and building your own frequency tables.
    Up the Irons
    What Would Jimi Do? Smash amps. Burn guitar. Take the groupies home.
    "Death Before Dishonour, my Friends!!" - Bruce D ickinson, Iron Maiden Aug 20, 2005 @ OzzFest
    Down with Sharon Osbourne

    "I wouldn't hire a butcher to fix my car. I also wouldn't hire a marketing firm to build my website." - Nilpo
  8. #5
  9. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2010
    Posts
    95
    Rep Power
    5
    Check these out too.

    Language Frequencies
    Letter Frequency Calculators

    The American Cryptogram Association is one of the only places you are likely to find info on foreign language crypts.
    Last edited by nik_meback; October 21st, 2011 at 04:43 PM.

IMN logo majestic logo threadwatch logo seochat tools logo