You would use a messaging solution in your enterprise application when you want to permit asynchronous processing, and you want to decouple a process from it’s implementation.
What are the pros & cons of using a messaging infrastructure
The pros of using a messaging infrastructure are that you gain asynchronous processing of your application, process persistence, and you are able to keep your applications and implementations loosely coupled. Thus, if one stage of your application workflow gets overloaded, the other stages upstream are able to continue to respond to requests from their clients without impact to these processes’ clients and service providers. Keeping your applications and implementations loosely coupled will allow you to be able to respond to changes in requirements, regulations, and environmental implementation changes without impact to the whole workflow. So, if you were to change the underlying algorithm, implementation language, or some other variable of one of your stages in your workflow, the clients and service providers for this stage of the workflow would remain unchanged, so long as you honor the original message contract.
The cons to a messaging solution are: Slower throughput than in-process communication, multiple layers of configuration, and many small sub-systems which require individual configurations and maintenance. Message based enterprise applications are usually distributed, so you gain all the headaches of network communication as well. Since all of these small applications need to be wired together just so, documentation is more important than with a typical monolithic application.
What has your experience been with similar architectures
If you need something robust, scalable, and redundant, messaging is the way to go. If you need something performant, messaging is still usable, but there will be a lot of optimization involved to get the level of performance out of it you need.
Is building your own messaging infrastructure recommended If yes, under what circumstances
There are enough open source and commercial messaging systems out there that handle persistence, and communication models for you that only under the most extreme or minor circumstances would I suggest building your own messaging infrastructure.
How does an ESB differ from a messaging infrastructure
An ESB sits on top of a messaging system, providing an abstraction without requiring one to bother with implementation specifics. With a messaging system, you are directly wiring your applications together, an ESB allows you to just subscribe to the ESB, and then configure workflow via some ESB configuration portal. This is the difference between coordination and orchestration.
I’ve included links to some resources which you may find useful.