MQ version 1.01 =========================== Need Perl 5.6.1 or later. Only tested fully on Win98, Win2000. Tested and works on Linux but not fully tested. COPYRIGHT AND LICENCE --------------------- You may distribute this work under the terms of either the GNU General Public License or the Artistic License, as specified in perl's README file. Copyright © 2002 Peter Tschipper Description ----------- A guaranteed delivery messaging system using send and receive queues. This is useful for asyncronously communicating between applications or simply to deliver information from one location to another in a safe way. For instance you could use it to monitor data step information coming in from several telephone switches that live in different NT domains and then generate CGI/HTML reports on the fly from the centrally gathered data. Send queues can read messages from a socket or from files that are dropped in the "Message_files" folder. Messages once read in by the send queue will then be either sent to the receive queue or, if the receive queue is currently unavailable, stored for later delivery. If the network or any system fails and is then restarted, each message and only one copy of each message is guaranteed to arrive at the correct destination (receive queue). You can run multiple send or receive queues on the same machine and you can also have multiple send queues that send messages all to the same receive queue. Setting up Send and Receive queues ---------------------------------- Send queues and receive queues can be easily setup with just three lines of code and a config file. See the examples "sendq.pl" and "receiveq.pl" contained in this distribution. The only thing that will change if you want create different queues is the path to the config file which is passed to the "start_queue()" method. And of course the contents of the config files will be slightly different for each queue. See section on config files. Once the code is written you can then install the queue as a service using the "srvany.exe" tool found in the NT resource kit. Testing the queues ------------------ You can get a send and receive queue up and running quickly by just launching the "send.pl" and "receive.pl" scripts included in this distribution. Once they are both running then you can send data through the system by running the "test_newline.pl" script. You should immediately start seeing data coming through to the receive queue. A folder will be created called "c:/perlmq" and in that folder will be two folders "ReceiveQueue1" and "SendQueue1". If you look in the "ReceiveQueue1" folder there will be a "Message_Files" folder. It is in the "Message_Files" folder that you will find text files that contain the tranmitted messages. As an experiment you can kill the receive queue process while messages are being sent. What will happen is that any new messages will now be stored in the "c:/PerlMQ/SendQueue1/Message_Files" folder. After a while launch the receive queue process again and you can see how the send queue is drained of pending messages. You can also put any text file in the "c:/PerlMQ/SendQueue1/Message_Files" folder and whatch it get transmitted to the receive queue - NOTE: each line in a file is considered a separate message and will be assigned it's own unique key to distinguish it from all other messages. Unique keys ----------- Each message that is sent from a send queue to a receive queue will have a unique key inserted at the beginning of the message followed by a "\t" character. This unique key is nessessary to ensure that no duplicate messages are sent and it's also a way of tracking when and where the messages came from because the key is a string composed of the time the message was sent as well as from what host and what queue they originated from. Receive queue message files --------------------------- The receive queue will store incoming messages in the "/Receive_queue/Message_files" folder. The message files are created in accordance with the "STORAGE" config file setting. This setting determines whether a new storage file is created Monthly, Daily, Hourly, Minutely etc. Once the messages are in the files they are considered to have been successfully delivered. It is up to the developer to then take those delivered message files and take whatever further actions are nessessary, such as: creating HTML reports; dumping the data into a database; etc... Setting up the queue monitoring ------------------------------- Queue monitoring is easy with the QueueMonitor.pm module. All you have to do is setup your config file for each queue (port, host, and message type) . See the QueueMonitor.conf supplied in thid distribution. You can also run the QueueMonitor.pl which will print out the status off all queues every 10 seconds. Currently you can only monitor the send queues and the only status you can get is if the queue is up/down and the number of messages files that have not been delivered yet (if there are message files you can be pretty sure the receive queue is down). The most important thing to remember when setting up your config file is to choose the right message type. This must be the same as that defined in the SendQueue.conf. If the send queue is reading from the socket using REC_SEP=\n then that's what you need to define in your QueueMonitor.conf for that queue. On the other hand if the send queue is reading messages using a LENGTH=4 then that's what you need to define in your QueueMonitor.conf for that queue. Config Files ------------ Each send and receive queue has an associated config file. You can name the files anthing that you want since the path to those files are passed as parameters upon starting up the queue, however, in this distribution you will find them named as SendQueue.conf and ReceiveQueue.conf. Both files in this distribution are setup in such a way that they should work as is if the send and receive queue are invoked on the same machine however in order to get them to work on different machines or have multiple queues running on one machine then you've got to make some changes. If you look in the config files the entries are fairly self explanitory however there are a few items to keep in mind: SendQueue.conf ------------- The only item that needs explanation are the REC_SEP and LENGTH values. These values define how any incoming messages are formatted when they are read from the send queue socket. Incoming messages can use either a record separator (REC_SEP) or a message length header (LENGTH) but not both defined at the same time. Record separators are usually newlines (\n) however you can use any sequence of non-escaped characters that you wish. Some escaped characters, such as, \n, \c, \a etc. are supported. For a complete listing of supported escape characters see the SendQueue.conf. If your incoming messages have length headers then you want to use the LENGTH value. This value will typically be assigned the number 4 and will indicate that the message length header is four bytes, or in other words, the first four bytes of each message will contain the length, in bytes, of the remaining message that follows. LENGTH can be set to any integer value but generally 4 is the norm. ReceiveQueue.conf ---------------- For each receive queue you have to select a unique RECEIVE_QUEUE_PORT and MESSAGE_FILES path. The only item that needs explanation is the STORAGE type. The STORAGE type defines what filenames are given to the stored message files. For instance if STORAGE=D then a new message file will be created every day and be of the format "YYYYMMDD.txt". If STORAGE=MIN then a new message file will be created every minute and be of the format "YYYYMMDDHHMI.txt". If STORAGE=U then each message will be placed in a unique file and the name of the file will be the "key" of that message. See the ReceiveQueue.conf for a more complete list of available formats. Things to watch out for - warnings/limitations ---------------------------------------------- - Of course, remember to make sure your port assignments are correct. - If you are using the QueueMonitor or want to your send queue to read messages from a socket make sure you are using the right record separator or message length. - Don't forget...Message files are not secure and can in some cases be read or deleted by others. Things that are coming ---------------------- - Use of more generic message addresses like "recipient.queue1@host" rather than having hard coded receive queue ports. - An application interface or module that will allow the user to put messages into the queue or take messages out of the queue. - Triggers for messages that are received. - Automatic message forwarding - Message queues that use a MySQL database to store messages. This will allow for better security. - Message Encryption? INSTALLATION ------------ To install this module type the following: perl Makefile.PL make make test make install DEPENDENCIES ------------ These modules requires perl 5.6.1