Queueing

RabbitMQ is used for queueing cache seeding and cache refreshes. There are separate RabbitMQ instances for each environment.

The administrative URL for the production RabbitMQ server used by the Refinery is:

http://rmq.aws.nypl.org:15672/

Each server node creates it's own queue and delayed queue with a name based on the hostname of the server. All server nodes in the environment share the same exchanges and messages are routed to the appropriate queue based on the routing key.

Queue and Exchange Configuration

There are two main queues used by the Refinery:

RefineryQueue: this is the "regular" queue read by consumers and used for processing messages.

RefineryQueueDelayed: this queue is used to "delay" messages via the dead letter exchange option. Messages put in this queue will NOT be consumed but, instead, will be re-published to the RefineryExchange upon message expiration (i.e. TTL).

There are two main exchanges used for publishing messages:

RefineryExchange: this is the "regular" exchange used for routing messages to the RefineryQueue.

RefineryExchangeDelayed: this is the exchange used for routing messages to the RefineryQueueDelayed.

Jenkins Jobs

There are Jenkins jobs (restart-refinery_mq) to START, STOP, or RESTART the queueing processes in each environment:

https://ci-sa.prod.aws.nypl.org/jenkins/search/?q=restart-refinery_mq

Publishers and Requeueing

Cache seed messages (with a type of InitializeCache) are initially published to the queue when the Jenkins server jobs are run with the START or RESTART option.

After these messages are consumed, they are added back to the delayed queue (with a TTL used as a "delay" specified by the configuration) essentially "re-queueing" them for later processing. This creates a continual loop of these messages.

Consumers

The Jenkins server jobs run with the START or RESTART option start consumers on each server node. The number of consumers is specified by the scripts/server_start.sh script (currently set at 1). 

These consumers consume messages from their respective RefineryQueue and remain running indefinitely on the server.

Troubleshooting

• Consumers should always be running on the server nodes to process queue messages. If a consumer stops running, the RESTART option should be run from the Jenkins job.