NServiceBus Message Replay Archive Architecture
I'm building an application that needs to preserve copies of the messages it is sending so that I can replay all the messages at a later stage. This is necessary because the processing of the messages will change dramatically over the cours开发者_Go百科e of development, but the data must be captured ASAP since it is realtime observation data. I can't seem to find any built-in functionality that directly addresses this and while I could write a custom tool to persist the data, that seems to contradict the purpose of using NServiceBus in the first place. Some options I'm considering:
Use the ForwardReceivedMessagesTo functionality of the Target bus to create an Archive queue, and build a simple application which uses this Archive queue as an input queue for simply forwarding messages onto the Target bus whenever the Replayer tool runs. This does clear the Archive queue, requiring it to be backed up first using the mqbkup utility, but this can be automated as part of the replay process. Alternatively, using two alternating Archive queues (one taking in new messages, and one for replaying) would solve this.
Use a publish/subscribe model and have an Archiver subscribe to the Target queue, placing the message in an Archive queue. A Replayer tool similar to the one above could use the Archive queue as an input queue and forward the messages to the Target. This would also clear the Archive queue, requiring one of the solutions above.
MassTransit people mention something called BusDriver that allows copying between queues but I can't find anything more about it.
My primary concern is to choose the approach that is least likely to lose data as once an observation is made it can never be made again outside of a narrow time window. This seems like it should be a common problem and yet I can't seem to find a straightforward solution to it. Suggestions?
Update I've decided to go with a journalled Target queue. I'll have an Archiver use the journal as an input and store messages to a database (could just be file-based), as well as allow for replaying messages from that database to the Target queue. While it would be possible to write a tool that copies messages from the journal queue to the target queue, the real problem - from a practical perspective - is that of managing the journal queue: it can't be backed up easily (mqbkup brings down the MSMQ service, which is unacceptable), and operating non-destructively on the queue requires me to write an MSMQ-based tool when I'd rather stick to the NServiceBus level of abstraction. Ultimately, MSMQ is a transport and not a store of messages so it needs to be treated as such.
The first option seems viable. I would add that NSB comes with a tool named ReturnToSourceQueue.exe that will replay messages for you. You could turn on journalling to keep messages, but something would have to clean that up since it doesn't roll. We have NetApp and we've used its SnapMirror to backup queue data, I'm sure there are similar things out there.
#3 BusDriver command-line reference:
BusDriver is a command-line utility used to administer queues, service bus instances and other things related to MassTransit.
Command-Line Reference
busdriver.exe [verb] [-option:value] [--switch]
help, --help Displays help
count Counts the number of messages in the specified
queue
-uri The URI of the queue
peek Displays the body of messages in the queue without
removing the messages
-uri The URI of the queue
-count The number of messages to display
move Move messages from one queue to another
-from The URI of the source queue
-to The URI of the destination queue
-count The number of messages to move
requeue Requeue messages from one queue to another
-uri The URI of the queue
-count The number of messages to move
save Save messages from a queue to a set of files
-uri The URI of the source queue
-file The name of the file to write to (will have .1, .2
appended automatically for each message)
-count The number of messages to save
--remove If set, the messages will be removed from the queue
load Load messages from a set of files into a queue
-uri The URI of the destination queue
-file The name of the file to read from (will have .1, .2
appended automatically for each message)
-count The number of messages to load
--remove If set, the message file will be removed once the
message has been loaded
trace Request a trace of messages that have been received
by a service bus
-uri The URI of the control bus for the service bus
instance
-count The number of messages to request
status Request a status probe of the bus at the endpoint
-uri The URI of the control bus for the service bus instance
exit, quit Exit the interactive console (run without arguments
to start the interactive console)
Examples:
count -uri:msmq://localhost/mt_server
Returns the number of messages that are present in the local
MSMQ private queue named "mt_server"
peek -uri:msmq://localhost/mt_client
Displays the body of the first message present in the local
MSMQ private queue named "mt_client"
trace -uri:msmq://localhost/mt_subscriptions
Requests and displays a trace of the last 100 messages received
by the mt_subscriptions (the default queue name used by the
subscription service, which is part of the RuntimeServices)
move -from:msmq://localhost/mt_server_error -to:msmq://localhost/mt_server
Moves one message from the mt_server_error queue to the mt_server
queue (typically done to reprocess a message that was previously
moved to the error queue due to a processing error, etc.)
精彩评论