> Is there any way to prevent reverse engineering on my firmware library and protect my IP.
How will you be delivering the library (source or object code)?
There are various ways of obfuscating things to make it more difficult to work out what is going on (though not impossible).
> What kind of agreement should I make with the customer?
At the very least, you'll need some kind of NDA
If you're looking at $10K++ worth of investment, then buying a couple of hours with a commercial lawyer should be something you should consider as well.
> Should I get a license fee per copy or should it be royality free?
Well that depends on what you want (and which you think will be the best earner for you).
Do you see high volume shipments of finished products? A few cents on 1M shipped units is a nice sum to be getting on with. But with it comes the need to verify actual sales in some way.
Or is it low volume to a specialised market, in which case you might look at a separate licence for each unique product line.