Chapter 11. The Queue

Table of Contents
11.1. Message Submission Areas
11.2. Router Behaviour on Queues
11.3. Scheduler, and Transport Agents
11.4. Postmaster Analysis Area
\fbox{\bf **Picture of queues**}
\medskip\hrule\medskip
\caption{ZMailer's queue structure}
\label{fig:adm:queues1}

There are multiple queues in ZMailer. Messages exist in in one of five locations:

and sometimes is also copied into the

11.1. Message Submission Areas

Message submission is done by writing a temporary file into the directory ($POSTOFFICE/public/), the actual format of the submitted message is described in Appendix D.

When the temporary file is completely written, it is renamed into one of the router input directories, usually into $POSTOFFICE/router/ ith a name that is a decimal representation of the spool-file i-node number. This is a way to ensure that the name of the file in the $POSTOFFICE/router/ directory is unique.

The message may also be renamed into alternate router directories, which give lower priorities on which messages to process when.

Sometimes, especially smtpserver built files may be moved into alternate directories. The smtpserver "ETRN" command has two implementations, original one is by moving the built special file to the directory $POSTOFFICE/transport/ without going through the router. The smtpserver may also move newly arrived files into the $POSTOFFICE/freezer/ directory.

11.2. Router Behaviour on Queues

FIXME:
   This description is from era before the router got
   ``daemonized'' in a sense of having separate instance
   of queue processor (and it also handles logging/log rotation)
   and a worker-farm of routing work processes.

FIXME: The system can have multiple router processes running in parallel and competeting on input files. Multiple processes may make sense when there are multiple processors in the system allowing running them in parallel, but also perhaps for handling cases where one process is handling routing of some large list, and other (hopefully) will get less costly jobs.

The router processes have a few different behaviours when they go over their input directories.

First of all, if there are ROUTERDIRS entries, those are scanned for processing after the primary $POSTOFFICE/router/ directory is found empty.

Within each directory, the router will sort files at first into mod-time order, and then process the oldest message first. (Unless the router has been started with the `-s' option.)

The router acquires a lock on the message (spool file) by means of renaming the file from its previous name to a name created with formatting statement:
  sprintf(buf, "%ld-%d", (long)filestat.st_ino, (int)getpid());

Once the router has been able to acquire a new name for the file, it starts off by creating a temporary file of router routing decisions. The file has a name created with formatting statement:
  sprintf(buf, "..%ld-%d", (long)filestat.st_ino, (int)getpid());

Once the processing has completed successfully, the original input file is moved into the directory $POSTOFFICE/queue/, and the router produced scheduler work-specification file is moved to the $POSTOFFICE/transports/ directory with the same name that the original file has.

If the routing analysis encountered problems, the message may end up moved into the directory $POSTOFFICE/deferred/, from which the command zmailer resubmit is needed to return the messages into processing (The router logs should be consulted for the reason why the message ended up in the /deferred/ area, especially if the command zmailer resubmit is not able to get the messages processed successfully and the files end up back in the /deferred/ area.)

If the original message had errors in its RFC-822 compliance, or some other detail, a copy of the message may end up in the directory $POSTOFFICE/postman/.

See Postmaster Analysis Area on section Section 11.4.

11.3. Scheduler, and Transport Agents

The scheduler work specification files are in the directory $POSTOFFICE/transport/, under which there can be (optionally) one or two levels of subdirectories into which the actual work files will be scattered to lessen the sizes of individual directories in which files reside, and thus to speed up the system implied directory lookups at the transport agents, when they open files, (and also in the scheduler).

When the router has completed message file processing, it places the resulting files into the top level directory of the scheduler; $POSTOFFICE/transport/, and $POSTOFFICE/queue/.

The scheduler (if so configured by ``-H'' option) will move the messages into ``hashed subdirectories,'' when it finds the new work specification files, and then start processing them.

The transport agents are run with their CWD in directory $POSTOFFICE/transport/, and they open files relative to that location. Actual message bodies, when needed, are opened with the path prefix ``../queue/'' to the work specification file name.

Usually it is the transport agent's duty to log different permanent status reports (failures, successes) into the end of the work-specification file. Sometimes the scheduler also logs something at the end of this file. All such operations are attempted without any sort of explicit locking, instead trusting the write(2) system call to behave in an atomic manner while writing to the disk file, and having a single buffer of data to write.

Once the scheduler has had all message recipient addresses processed by the transport agents, it will handle possible diagnostics on the message, and finally it will remove the original spool-file from the $POSTOFFICE/queue/, and the work-specification file from $POSTOFFICE/transport/.

11.4. Postmaster Analysis Area

If the filename in the $POSTOFFICE/postman/ directory has an underscore in it, the reason for the copy is soft, that is, the message has been sent through successfully in spite of being copied into the directory.

If the filename in that directory does not have an underscrore in it, that file has not been processed successfully, and the only copy of the message is now in that directory!

Usually forementioned underscoreless filenames

If the smtpserver receives a message with content that the policy filtering system decides to be dubious, it can move the message into $POSTOFFICE/freezer/ directory with a bit explanatory name of type:
  sprintf(buf, "%ld.%s", (long)filestat.st_ino, causecodestring);

The files in the freezer-area are in the input format to the router, and as of this wiring, there are no tools to automatically process them for obvious spams, and just those that were falsely triggered.