Broker¶
The Broker is the part of Comet which receives messages from authors and distributes them to subscribers. It can also subscribe to other brokers and act upon events received.
Twisted Applications¶
The Comet broker is implemented as a Twisted application. This means that it is not invoked directly
on the command line, but rather using the twistd
tool provided as part
of Twisted.
twistd
provides a wide range of generic configuration options related to
daemonizing, logging, profiling, and so on. This fall outside the scope of
this documentation, but the interested reader is encouraged to familiarize
themselves with the contents of twistd(1)
.
It is worth noting that, by default, processes invoked with twistd
are
immediately daemonized. That is, twistd
is invoked, it returns control to
your shell prompt almost immediately, but Comet is now running in the
background. This is generally very convenient, but sometimes it’s useful to
have the application run in the foreground when testing and debugging: to
achieve this, invoke twistd -n
.
Invoking Comet¶
Specify name of the Twisted application to run after twistd
and its
options. In this case, we run comet
, and pass it the --help
option to
provide a brief usage message:
$ twistd comet --help
Usage: twistd [options] comet [options]
Options:
-r, --receive Listen for TCP connections from authors.
-b, --broadcast Re-broadcast VOEvents received.
-v, --verbose Increase verbosity.
-q, --quiet Decrease verbosity.
--print-event Enable the print-event plugin.
--save-event Enable the save-event plugin.
--local-ivo= IVOA identifier for this system (required for
--receive and --broadcast)
--eventdb= Event database root. [default: system dependent]
--receive-port= TCP port for receiving events. [default: 8098]
--broadcast-port= TCP port for broadcasting events. [default:
8099]
--broadcast-test-interval= Interval between test event brodcasts (in
seconds; 0 to disable). [default: 3600]
--author-whitelist= Network to be included in author whitelist.
[default: 0.0.0.0/0]
--subscriber-whitelist= Network to be included in subscriber
whitelist. [default: 0.0.0.0/0]
--remote= Remote broadcaster to subscribe to
(host[:port]).
--filter= XPath filter applied to events broadcast by
remote.
--cmd= Spawn external command on event receipt.
--save-event-directory= Target directory [default:
current working directory]
--version Display Twisted version and exit.
--help Display this help and exit.
Basic Modes of Operation¶
If the --receive
option is supplied, Comet will fulfil the Broker role in
an Author to Broker connection. In other words, it will listen for TCP
connections from remote authors and accept events for distribution. The TCP
port on which Comet will listen may be specified with the --receive-port
option.
If the --broadcast
option is supplied, Comet will listen for Subscribers
to connect and then it will fulfil the Broker role in a Broker to Subscriber
connection with each of the Subscribers. Any VOEvents received (either by
direct connection from authors, or by subscribing to remote brokers) are
rebroadcast to subscribers. The TCP port on which Comet will allow subscribers
to connect may be specified with the --broadcast-port
option.
If one or more --remote
options are supplied, Comet will subscribe to the
remote host specified and fulfil the Subscriber role in the resulting Broker
to Subscriber connection. If just given a hostname Comet will attempt to
subscribe on port 8099. Optionally, a different port may be specified by
appending it to the hostname, separated by a :
.
A single Comet daemon will accept any combination of --receiver
,
--broadcast
and one or more --remote
options and play all of the
specified roles simultaneously. If none of --receiver
, --broadcast
or
--remote
are supplied, there is no work to be done and Comet will exit
immediately.
Identification¶
Comet identifies itself to other systems by means of an International Virtual Observatory Identifier or IVOID: see the IVOA Identifiers Version 2.0 standard for details.
Note
The VOEvent standard uses the older term “IVORN” rather than IVOID; Comet prefers the more modern usage.
You should specify some appropriate IVOID for your site using the
--local-ivo
option. This is required when one or both of --receive
or
--broadcast
is specified; it’s optional, but will be used if provided,
otherwise. This should be of the format ivo://${organization}/${name}
;
for example, ivo://org.transientskp/comet_broker
.
VOEvent Network Maintenance¶
In order to prevent looping on the network (ie, two brokers exchanging the
same event ad infinitum), a database of previously seen event is maintained.
This database is written to the filesystem in the location specified by the
--eventdb
option. Events which are recorded in the database are not
forwarded by Comet. This is important: looping would degrade the quality of
the VOEvent network for all users! Note that events persist in the database
for 30 days, after which they are expired to save space.
Receiver Options¶
When acting as a receiving broker (with --receive
), Comet will only accept
new events for publication from hosts which have been specified as
“whitelisted”. Hosts (or, indeed, networks) may be included in the whitelist
using the --author-whitelist
option. This option accepts either CIDR or dot-decimal notation
including a subnet mask. For example, --author-whitelist 127.0.0.1/32
and
--author-whitelist 127.0.0.1/255.255.255.255
both permit the local host to
submit events to the broker. This option may be specified multiple times and
the results are cumulative. To accept submissions from any host, specify
--author-whitelist 0.0.0.0/0
; this is the default if no
--author-whitelist
option is supplied.
Broadcaster Options¶
By default, Comet will broadcast a content-free test event to all subscribers
every hour. The aim is to help with network debugging. The interval between
test events may be configured using the --broadcast-test-interval
option,
which accepts a value in seconds. Set it to 0
to disable the test
broadcast completely.
The broadcaster supports a similar whitelisting feature to the receiver: in
this case, new subscribers to the broadcast will not be accepted unless their
IP address has been added to the whitelist. This is enabled using the
--subscriber-whitelist
option; its semantics are the same as
--author-whitelist
.
Subscriber Options¶
When subscribing to a remote broker (with --remote
), one or more filters
may be specified which limit the events which will be received. These filters
are specified with --filter
, in the form of XPath 1.0 expressions. The broker will evaluate the
expression against each event it processes, and only forward the event to the
subscriber if it produces a non-empty result. For more details see the section
on filtering.
Common Options¶
Plugins¶
Custom code may be run to perform arbitrary local processing on an event when
it is received. For more details, see the section on event handlers. Plugin actions will be taken whether Comet receives an event
from an author (--receive
) or an upstream broker (--remote
). A plugin
is enabled by giving its name as a command line option (--plugin-name
).
Plugins may also take arguments from the command line. These are given in the
form --plugin-name-argument=value
.
Comet ships with two plugins which both serve as examples of how to write
event handlers and which may be useful in their own right. The first simply
writes events to Comet’s log as they are received. This is the print-event
plugin: enable it by invoking Comet with the --print-event
option.
The second plugin shipped with Comet is save-event
, which writes events to
file. It is enabled with the --save-event
option. By default, events are
written to the default working directory (normally the directory in which you
invoked Comet): this may be customized using the --save-event-directory=
option. The filename under which an event is saved is based on its IVOID, but
modified to avoid characters which are awkard to work with on standard
filesystems.
Spawning External Commands¶
Similarly, received events may be sent to one or more external commands for
processing. These are specified using the --cmd
option. They should accept
the event on standard input and perform whatever processing is required before
exiting. The standard output and error from the external process will be
written to Comet’s log with level DEBUG
. If it returns a value other than
0, it will be logged as a failure. Note that external commands are run in a
separate thread, so will not block the subscriber from processing new events;
however, the user is nevertheless responsible for ensuring that they terminate
in a timely fashion.
Logging¶
The amount of information Comet writes to its log may be adjusted using the
--verbose
and --quiet
options.