October 27, 2004

Programs, particularly those in the enterprise, cannot stand alone any longer. Whether needing to exchange data or provide some kind of service-like behavior in a centralized (or distributed) fashion, programs need to talk to other programs. This form of communication between programs, often called inter-process communication, or IPC, can take many forms and styles, each with its own inherent advantages and disadvantages.

One approach is to exchange files via a network share, clearly what might be the simplest approach; on the other hand, a custom socket driven approach causes more implementation effort but yields a much tighter-grained control over how the data is exchanged. Some of the most popular approaches today include .NET Remoting and, especially between platforms, Web Services. These are popular partly because they come with a base set of features and a rich programming interface. Unfortunately, both require near-constant access between clients and servers in order to work. In contrast, message servers offer a full featured package for managing and monitoring messages that guarantees the delivery even if the destination system is temporarily unavailable.

A popular example for the use of message servers is an order entry system, e.g. an online shop. When the user finalizes the shopping cart, the system has to process the order and display a "Thank you!" page for confirmation. Instead of processing the order in real-time, a message-based approach can defer the actual processing to a different program, thus deferring the actual work (and the wait time incurred as a result) to a later time. The online shop only sends the order as a message to its backend system and therewith its job is done and the final "Thank you!" page can be displayed. Meanwhile the backend system monitors the incoming orders and starts the fulfilment process asynchronously.

Obviously messaging needs to be absolutely reliable and that's one of the key features that message servers provide. Moreover, message servers are able to build routing networks where a message can be routed through several servers to its destination system. That allows you to create a high available network of backend systems.

Microsoft's Message Queuing (MSMQ) is such a message server and is part of the Windows Server operating system. Originally MSMQ came with a native, COM-based programming interface. But Microsoft has added a.NET wrapper on top of it which makes the development of MSMQ-based systems quite easier. The first version of MSMQ was delivered with Windows NT 4.0; now Windows XP/2003 brings the newest version (3.0), provding several interesting new features. Given that scalability is a large reason for using message-oriented middleware, it's really time to have a closer look on how to use MSMQ from a developer's perspective.

It's like emailing!

Messaging is can be seen as an email system for applications. Instead of inboxes, MSMQ uses queues for storing messages. Queues are also persistent - if queuing service is down, e.g. for maintenance, messages are still available when the service is back again. Like email folders, queues have their own unique addresses. Finally, and perhaps most importantly, the content of a message is not specified. It can be a simple text based "Hello World!" text or more complex things like objects, as long as they can be serialized.

The flow of messaging (image 1) illustrates how messaging works. Please keep in mind that the receiving application can be equal to the sender or a quite different system which has access rights. Moreover, send and receive operations will be typically done asynchronously. Step 4 in the draft allows two different options: a) the application pulls new messages or b) MSMQ pushes messages to destination system usually done by the MSMQ trigger mechanism.

Image 1: Draft of Messaging

A quick start sample

As a software developer I prefer to have a look at some code first before studying tons of whitepapers and API documentation -- as you know, a code-snippet says more than thousand words. Here's a quick start console application:

// Very simple quickstart sample
  using System;
  using System.Messaging;
  class MsmqQuickstart {

  	[STAThread]
  	static void Main(string[] args) {

  		// (1) path to message queue
  		// syntax: [computer name]\private$\[queue name]
  		// use "." for localhost
  		string queuePath = @".\private$\TSS_MsmqTest";

  		// (2) open message queue, create new one if not exists
  		MessageQueue messageQueue;
  		if (MessageQueue.Exists(queuePath))
  			messageQueue = new MessageQueue(queuePath);
  		else
  			messageQueue = MessageQueue.Create(queuePath);

  		// (3) send simple message
  		messageQueue.Send("Hello World!");

  		// (4) break here and have a lool in your message queue!!
  		System.Diagnostics.Debugger.Break();

  		// (5) receive message from message queue
  		Message message = messageQueue.Receive();
  		string myMessage = Convert.ToString(message.Body);
  		Console.WriteLine(myMessage);

  		// (6) close message queue
  		messageQueue.Close();
  	}
  }

In order to run this code MSMQ must be installed on the machine you are pointing to (MSMQ is not installed by default) and a reference to System.Messaging.dll must be added to the project. Because MSMQ is part of the Windows operating system, installation is part of the Windows setup. Please follow Add/Remove Programs, Add/Remove Windows Components and then select Message Queuing Services.

As you can imagine this application creates a message queue first (1), then sends a "Hello World!" message into the queue (2) and at last receives sent message back from the queue (4). In debug mode a programmatic break stops the application (3) and you will be able to see the message in Message Queuing administration console (Image 1).

Image 2: Hello World message is arrived

This quick start sample gives you a first feeling of what programming the message queue is like. In the following the articles goes through most important issues on MSMQ programming step by step: queue management, sending and receiving messages and transactions.

Message Queue Management

Message queue administration can be done via the console plug-in for MSMQ or using the managed API. Queue administration consists all relevant operations like creating and deleting queues and setting permissions. MSMQ has several different queue types: The two major types are private and public queues. A private queue is registered on the local machine that typically cannot be located by other applications while a public queue is registered in the directory service that can be located by any Message Queuing application. Journal queues can be enabled optionally in order to let the system store removed messages in. Dead letter queues are designed for undeliverable messages. That could be the case whenever a message cannot be delivered before its time-to-reach-queue timer expires. A queue can be addressed in several ways: using a path ("MyComputer\MyPublicQueue"), a format name ("FormatName:Public={guid-of-the-queue"} or by a specified label ("Label:MyQueueLabel"). Please find more on queue addressing here. Table 1 gives more information on access syntax using the path notation.

Table 1: Queue types and access syntax.

* use "." for local machine, e.g. ".\Private$\MyQueue"

The most important class for queue administration is System.Messaging.MessageQueue with its methods Create(), Delete(), Exists(), Purge() and SetPermissions(). The quick start sample above still uses the Exists() and Create() method in order to create a new queue. The next code snippet demonstrates the usage of theses methods:

// Sample code for queue administration
void Foo() {
	// (1) path to message queue
	string queuePath = @".\private$\TSS_MsmqTest_QueueAdministration";

	// (2) create new queue
	MessageQueue.Create(queuePath);

	// (3) check if queue exists
	bool queueExists = MessageQueue.Exists(queuePath);

	// (4) remove all messages from queue
	MessageQueue messageQueueA = new MessageQueue(queuePath);
	messageQueueA.Purge();
	messageQueueA.Close();

	// (5) delete existing message queue
	MessageQueue.Delete(queuePath);

	// (6) set permissions to message queue
	MessageQueue messageQueueB = new MessageQueue(queuePath);
	messageQueueB.SetPermissions("Donald Duck",
			MessageQueueAccessRights.ReceiveMessage);
	messageQueueB.Close();
}

Steps 4 and 6, purging messages and setting permissions, are working on an instantiated message queue. Therefore it's necessary to initialize a MessageQueue object first by specifying the queue as constructor parameter. Setting permissions can be still more complex than this smart call here. MSMQ supports an Access Control List concept (System.Messaging.AccessControlList) where detailed rights on a queue can be specified. The sample illustrates only one of four parameter combinations defined by SetPermissions().

Sending Messages

The quick start sample shows in a very simple way how a message can be sent to a queue using the Send() command of an initialized MessageQueue object. Maybe someone wonders where the Xml structure comes from as displayed in the screenshot (image 1) before and why application receives a message object instead of simple sent "Hello World!" string. The reason for that is that all messages will be put in a message envelope represented by the class System.Messaging.Message. The content of the message - "the real message" - will be serialized and put into the body property of that message object -- still analogue to emailing. The default formatter for serialization is XmlMessageFormatter and that's why the "Hello World!" message is wrapped into an xml envelope. Even if no message object has been created, as in the quick start sample, MSMQ creates a message object automatically. Furthermore, messages can be more than string-based messages because the body property of the message object is able to store any serializable object.

The Message Object

The message object itself contains several meta information on the message, such as timeouts, formatter to use for body part, priority, a digital signature or even an encryption algorithm. Some fields will be set at run-time by MSMQ and therefore are read-only at design time, for instance the timestamp when message was sent or sender's id. If you need control about these writable fields, you have to create a message object first, adjust it properties, set the body content and then pass it to the message queue. However, when application receives a message from a queue, always a message object will be returned and the embedded object in the body has to be casted back to its origin type.

Table 2: Properties of Message class (taken from .NET Framework documentation)

Dead Letter Queue and Acknowledge Types

At this point two options should briefly be explained: usage of dead-letter queues and concept of administration queues. As mentioned before, timeouts can specify in what timeframe a message must be received (TimeToBeReceived message property) or the maximum of time for reaching the destination queue in a complex multi-routing environment (TimeToReachQueue message property). If designated time out is reached, message will be deleted from queue. The message sender won't be informed about that! But by setting the UseDeadLetterQueue property to true, the message will be moved there. However, sender still won't be informed but the application is able to find all undeliverable messages in this queue. If application needs more feedback on status of message delivery, administration queues and acknowledge types are playing a big role. First step is to create an administration queue where MSMQ can store status messages to. The AcknowledgeType property specifies what MSMQ should report. There are several switches allowing you to get a detailed report message on the delivery status of messages. The following sample sets a timeout when a message should be received and also specifies the acknowledge type. Moreover, the UseDeadLetterQueue property is set, so undeliverable messages will also be stored there. (Note: Administration queue must be created first.)

Receiving Messages

In order to receive a message from queue the Receive() method of an initialized MessageQueue object must be called. Receive() is a synchronous method which blocks execution if no message exists in given queue. To avoid this behavior, a timeout can be specified as TimeSpan parameter. If this timeout occurs, a MessageQueueException will be raised. While Receive() removes the message from the queue, the Peek() method fetches messages but does not remove them. Using the GetAllMesages() command retrieves all existing messages from queue as an array of message objects also without deleting them. Moreover, MessageQueue class offers methods to retrieve lists of existing public and private queues, e.g. the GetPublicQueues() method.

The following sample summarizes the send and receive handling of messages:

using System;
using System.Messaging;

public class Person {
	public string Firstname = string.Empty;
	public string Lastname = string.Empty;

	public Person() {
	}

	public Person(string firstname, string lastname) {
		Firstname = firstname;
		Lastname = lastname;
	}
}

class MsmqObjects {
	[STAThread]
	static void Main(string[] args) {
		string queuePath = @".\private$\TSS_MsmqTest_ComplexMessages";

		// (1) open message queue, create new one if not exists
		MessageQueue messageQueue;
		if (MessageQueue.Exists(queuePath))
			messageQueue = new MessageQueue(queuePath);
		else
			messageQueue = MessageQueue.Create(queuePath);

		// (2a) pass person object directly to message queue
		Person person = new Person("Donald", "Duck");
		messageQueue.Send(person);

		// (2b) create message object first, enable dead letter queue,
		// assign person object and send message
		Message message = new Message();
		message.UseDeadLetterQueue = true;
		message.Body = new Person("Mickey", "Mouse");
		messageQueue.Send(message);

		// (3) Receive all messages stored in queue without removing
		Message[] messages = messageQueue.GetAllMessages();

		// (4) Just an example on timeout and its exception ...
		messageQueue.Purge(); // removes all messages
		try {
			// Because queue is empty, receive throws an exception 
                     after 4 sec.
			Message lastMessage = messageQueue.Receive(new
                                                          TimeSpan(0,0,4));
		}
		catch (MessageQueueException ex) {
			Console.WriteLine(ex.Message);
		}

		// (5) Close message queue
		messageQueue.Close();

	}
}

Using Transactions

The handling of transactions is done by special transactional queues. In order to create such a queue, the "transactional" parameter of the Create() method must be set to true. A transactional queue can only contain messages sent in a transactional context. The handling is very similar to SqlTransaction as next sample shows:

using System;
using System.Messaging;

class Transactions {
	[STAThread]
	static void Main(string[] args) {
		string queuePath_A = ".\private$\TSS_MsmqTest_Transactional_A";
		string queuePath_B = ".\private$\TSS_MsmqTest_Transactional_B";

		// (1) Create transactional queues
		MessageQueue messageQueue_A = GetMessageQueue(queuePath_A);
		MessageQueue messageQueue_B = GetMessageQueue(queuePath_B);

		// (2) Create transaction object
		MessageQueueTransaction msmqTransaction = new
                                                MessageQueueTransaction();
		try {
			// (3) Begin transaction
			messageQueueTransaction.Begin();

			// (4) Send messages two queues A and B
			messageQueue_A.Send("A.1", msmqTransaction);
			messageQueue_A.Send("A.2", msmqTransaction);
			messageQueue_A.Send("A.3", msmqTransaction);

			messageQueue_B.Send("B.1", msmqTransaction);
			messageQueue_B.Send("B.2", msmqTransaction);

			// (5) Commit transaction
			msmqTransaction.Commit();
		}
		catch {
			// (6) Abort transaction if an error occurs
			msmqTransaction.Abort();
		}

		// (7) Abort transaction if an error occurs
		messageQueue_A.Close();
		messageQueue_B.Close();
	}

	private static MessageQueue GetMessageQueue(string queuePath) {
		MessageQueue messageQueue;
		if (MessageQueue.Exists(queuePath))
			messageQueue = new MessageQueue(queuePath);
		else
			// parameter "true" -> Transactional
			messageQueue = MessageQueue.Create(queuePath, true);
		return messageQueue;
	}
}

This sample creates two transactional queues first (1) by setting the transactional property to "true". Then a MessageQueueTransaction object will be created (2) and its Begin() method called (3). The following send operations are using this transaction object (4) and all messages sent to queue A and queue B are using a single transactional context. If only one operation should fail, the error will be caught and transaction aborted (6). Is this case no message will be stored either in queue A nor in queue B. Otherwise, if no error raises, the Commit() command (5) lets MSMQ store all messages of this transaction.
This is a classical cross-queue transaction but it also runs on a single queue. Then MSMQ writes all messages of the transaction as a batch in the queue and keeps the order of messages even if another process writes messages in parallel to same queue. Analog to this send sample, transactions can be used for receiving. For instance: the received message should be stored in a local database. Then you'll commit the receive operation first when commit of database succeeded.

Table 3: Properties and Methods of MessageQueue class (cutout taken from .NET Framework documentation)

MSMQ 3.0 -- where are you?

As mentioned before, the new version 3.0 is available and it comes with some nice new features like message delivery over HTTP and the support of message distribution lists. A list of all new and enhanced features can be found at Microsoft's MSMQ page. That's the good news but there are bad news as well: these features are not supported by current .NET Framework 1.1 messaging classes. You are able to use MSMQ 3.0 in Windows XP/2003 but you cannot use the new features. In short, there is no "PeekByLookupId()" method today. But as you can imagine, upcoming .NET Framework 2.0 fully supports version 3.0. In the meantime you have the choice to use the COM wrapper by adding a reference to Microsoft Message Queue Object Library 3.0 (mqoa.dll). Visual Studio creates a wrapper for you, so you don't have to use PInvoke but the API is still very technical. Obviously, if you really, really need one of the new 3.0 features and you really can't wait for Whidbey to ship, the COM-wrapped approach is clearly the best way to go, but we don't promise it'll be easy. However, a short demo project using the COM wrapper is attached to the samples of the article.

Tell me, is there more inside?

This article has shown the fundamentals of MSMQ programming like storing and retrieving messages and basic management functionality like creating and deleting queues. Using the .NET API is very comfortable and intuitive. But working with MSMQ in a real-life project needs a basic understanding of messaging as architectural concept. In contrast to an RPC-based communication, no "methods" will be called. Systems become more loosely coupled and therefore more autonomous. Remember that a message is just a message and not an internal business object. Therefore I suggest to design new distributed applications following the base idea of service-oriented architecture by defining explicit boundaries and creating autonomous "services" and let MSMQ do the communication part.

Moreover, a powerful technology like MSMQ comes with a lot of complexity like different features depending on underlying installation platform, for instance the GetAllMessage() method is not available in a Windows workgroup mode on remote computers. This article gives you an introduction on programming and now it's on you to do the message queue!

Links

• Microsoft Message Queuing Center
  http://www.microsoft.com/windows2000/technologies/communications/msmq/default.asp
• MSDN Documentation System.Messaging
  http://msdn.microsoft.com/library/en-us/cpref/html/frlrfSystemMessaging.asp

Sebastian Weber is Software Engineer at Platinion GmbH in Germany, A Company of The Boston Consulting Group. He's specialized in building .NET-based applications and known as author and speaker in the field of .NET and Microsoft Server technologies. Sebastian can be contacted via http://weblogs.asp.net/SebastianWeber.

Authors

Sebastian Weber is Software Engineer at Platinion GmbH in Germany, A Company of The Boston Consulting Group. He's specialized in building .NET-based applications. Besides this, he's known as author and speaker in the field of .NET and Microsoft Server technologies. Sebastian can be contacted via http://weblogs.asp.net/SebastianWeber.