I have the following code ( generated by IDA )
A bit stripped. The &RC4Key field is actually part of an object.
Code:
    void MakeKeyData(char *lpszKeyString) {
      md5 = EVP_md5();
      EVP_Digest(lpszKeyString, strlen(lpszKeyString), pKeyData, 0, md5, 0);
      RC4_set_key(&RC4Key, 16, pKeyData);
    }
So, as far as i can tell, the code generates the MD5 hash of the lpszKeyString and uses that hash as a key for further RC4 encyptions.

Example :

Code:
    key        = 342B5480-2A71-430A-ABDA-E2D8363AA878
    MD5( key ) = A0 F7 AB 72 B0 E5 93 4D A3 D7 FD 2C E5 25 AE A0 
    
    ENCRYPTED  = CB 6F 55 26 77 05 0D 3D 55 DA 85 12 76 9B 57 01 AC 2F 2B 45 99 
    DECRYPTED  = FB 63 BA DC B4 D9 DC 87 E5 FD C0 F9 1F 3E BE D5 13 93 A2 44 2F 
    EXPECTED   = DA 01 4A 49 80 AD 67 45 C0 62 95 49
The expected line only has the first 12 bytes of the data, since the rest are harder to reproduce.

The RC4 algorithm i use is the same as OpenSSL uses : http://pastebin.com/xLQz7pww
I tested the algorithm on several inputs/keys and it works just fine.
Based on this : http://pastebin.com/Mu1CHSTS


Where did i go wrong ?