As other answers have suggested, VBA security in Excel 2003 (.xls) is flawed; a password is only a little annoyance, most often than not for the developper himself. IMHO you're absolutely better off with VSTO, even if VBA security in Excel 2007+ (.xlsx) is much better, not to mention the VBA IDE which hasn't seen improvements in centuries (although no so true with the right VBE add-in...).
A solution could be to write the sensible code in a separate library which your VSTO project calls; instead of intricate formulas you could expose functions which would appear as "black boxes" (i.e. an abstraction of the formula's functionality); how it's implemented is just not available to anyone without the source code), and without the dll the workbook couldn't be calculated (would be #NAME? all over the place).
For example, instead of =some proprietary formula returning a Smurf, users would see =Smurf(SomeTableColumnName, SomeRangeName, SomeRangeReference, SomeCellReference). Of course this comes at a performance cost, so I would only do this for stuff that's really important to protect.
I think a user that manages to extract the proprietary formulas from such an on-the-side library, deserves to leave with them - between that and Alt+F11, there's quite a step; the proprietary code is as "protected" as any other .net assembly is.
UPDATE
Still without using any 3rd-party libraries, if the proprietary code does not need Excel interoperability, a perhaps better solution could be to create the function in VBA (i.e. leave the Excel interop part within Excel) and only pass primitive types (VBA Integer => .net Int16; VBA Long => .net Int32, strings, etc.) to a COM-visible library which doesn't even need to use Excel interop; it will be COM interop instead of Excel interop and the performance hit would be significantly reduced, though still present (it remains .net managed code "talking" to unmanaged COM).
The VBA code's responsibility would be to read and write to/from the worksheets, and the COM-visible library's role would be to implement the calculations to be made out of the values passed by the VBA code; of course the VBA code would be easily accessible to the user (Alt+F11), but again they couldn't access the actual calculation implementation. And then the VBA code could be password-protected, and the calculation results without the DLL would be either 0's or #VALUE!, depending on how the VBA code is implemented.
The key is to leave Excel-specifics in Excel and have the .net library pick up where it can without referencing Excel interop assemblies - for example, the VBA function would take a cell reference, grab its value (perhaps validate it) and pass it to the .net assembly, which would return another primitive type that VBA would return to Excel.
Another plus is that the library code with the proprietary calculations could be reused, if you ever were to "convert" the Excel workbook into, say, a Web application.
