Batch User Workload Monitoring

proc_box is an open source, way of containerizing a process into a resource monitored box suitable for batch systems. The primary goal is to limit resource explosion and causing other users of the batch system to receive unexpected resource pressure. This project also provides remote control of the process through AMQP JSON messages. Resource usage measurements of the contained process are emitted for analysis or control by other tools.

Features

• Connects to an AMQP broker
• Starts the contained process
• Gracefully exits with either a simple remote control JSON or the process naturally existing.
• Basic suspend/resume/quit/kill remote control support.
• Controllable wall-clock timer to end runaway processes (also controlled over AMQP).
• Captures SIGINT/etc. at proc_box level and issues appropriate signals to child process.
• Periodic statistical samples of process usage (aggregated parent and children) emitted on AMQP.
• Arguments directly on the command line or provided through environment variables. Command line overrides environment variables.

Requirements

Go; that's about it. We suggest the rabbitmq:3-management Docker container for development purposes.

Documentation

Use Godoc documentation under the agents package for reference.

Running

Generally, only specifying both the AMQP broker and the command to be contained is necessary.

./proc_box -uri <amqp_uri> <cmd> <args>

Remote Commands

Remote commands can be issued through the AMQP broker to the topic exchange specified (or default proc_box.remote_control). Messages should be in JSON of the form:

{
    "command": "<cmd>",
    "arguments": ["<arg1>", "<arg2>", ...],
}

Supported commands:

• stop: Issue SIGQUIT to the process.
• resume: Resume suspended process execution (and timeout timer).
• suspend: Suspend the process execution.
• kill: Issue SIGKILL to the process. Optional single argument is the signal to send (9 is default).
• sample: Force a process statistics sample.
• change_sample_rate: Change duration between process statistic samples.
• timer_reset: Disables timer and sets ElapsedTime to zero.
• timer_start: Enables timer ticking from an ElapsedTime of zero.
• timer_stop: Disables timer
• timer_resume: Continues ticking time without resetting.

License

proc_box is licensed under the MIT License.