The elephant in the room is that Lisp's community makes it very hard to adopt Lisp for production. Scheme is not as affected, but in general you end up with several implementations, each doing now-common things like networking slightly differently, and each having a feature set that contains 90% of what you want. What's worse, if you get into non-trivial things it can be difficult to migrate from one lisp implementation to another when your priorities change and different features are required.
Compare this to most other languages–notable outliers like C/C++ excepted–have canonical implementations that world-wide communities can germinate from. This additional confusion at the outset of adoption has stopped more than one of my colleagues from learning Lisp.
There's also typically deployment issues. Because most lisp execution environments are so heavily tweaked out, a lot of times it's not clear how to deploy a piece of software. A great example of this was a web-app I received from a talented lisp consultant. His method for running the program? "Run SLIME in a Screen session and type the following commands." You can rightly argue that this was just one bad example, but I think everyone who's being honest has seen this kind of crap going on, and a clearer and more obvious way to deploy would do most lisps a lot of good.
