Many data reduction tasks involve software running in several processes on one computer, or on several processes on each of many computers. This web service allows the end user to keep track of the progress of each 'worker', by displaying a bar chart showing the live status of each worker. The workers send their status to the server with a single HTTP request - that could be a single line calling 'curl http://....' in a shell script, or a requests.get('http://...') function call in a Python script. The web service collates the updates from all of the workers, and allows the users to view the status of all workers, in real time, in a web browser.

Eg:  http://ws.mwatelescope.org/progress/show?jobid=demo1

To use this service, whatever starts all the worker processes needs to use the 'create' service to define a new job, with a job ID that's unique. The job ID is an arbitrary string or number - possibilities include a random 32-bit hash or integer, a stirng made up of the observation ID plus the current Unix timestamp, etc. The job ID distinguishes this multi-process job from any other job, run by you or someone else. Job data is stored on the server for at least a week (by default).

After the progress job has been created by calling a /progress/create URL, the startup code than has to pass the job ID to all of the worker processes (along with the data that they need to process), and send the user the job ID that it chose, or the 'view plot' URL with that job ID.

Each of the worker processes then calls a '/progress/update' URL at regular intervals to send its current state to the web service. It sends the job ID that it's been given, its own 'worker ID' string (unique inside that job), the total number of 'things' that it's been told to process, and the number it's done so far. This is simply plotted as (100 * current / total), as a percentage complete. Workers can also include an arbitrary string (shown in the plot inside or next to their progress bar), and an 'ok' flag, which if equal to zero, results in their bar being plotted in red instead of green.

The web services themselves:

The URLs for these services all have a similar format, starting with the base URL, e.g.

http://ws.mwatelescope.org/

Parameters are passed in the URL - a '?' character separates the path from the first 'name=value' parameter, and '&' characters separate any subsequent 'name=value' parameters.

create - make a new progress job.

BASEURL/progress/create - eg http://ws.mwatelescope.org/progress/create?jobid=myjob123&desc=Testing

Parameters are:

The create service is called once, before any workers are started. The job ID used then needs to be passed on to the worker processes, as they are started, and sent to the end user, so they can view the progress bar in a web browser. If you call the create service with a job ID that already exists in the system, the existing job will be overwritten, and any existing progress data from workers will be lost (but if those workers are still running, it will get any subsequent status updates).

update - set the current state of a worker process.

BASEURL/progress/update - eg https://ws.mwatelescope.org/progress/update?jobid=myjob123&workid=1&current=5&total=22, https://ws.mwatelescope.org/progress/update?jobid=myjob123&workid=2&current=10&total=22, https://ws.mwatelescope.org/progress/update?jobid=myjob123&workid=3&current=21&total=22

Parameters are:

The update service should be called at regular intervals by each worker, to send their current state to the web service.

show - display an animated progress bar plot for this job.

BASEURL/progress/show - eg https://ws.mwatelescope.org/progress/show?jobid=myjob123

Parameters are:

This service returns a web page showing a dynamic (automatically refreshes) plot of the current state of each of the worker processes for that job. 

render - return a PNG file with the progress bar plot for this job.

BASEURL/progress/render - eg https://ws.mwatelescope.org/progress/render?jobid=myjob123

Parameters are:

This service returns a single PNG file with the plot for a given. Useful if you want to embed the progress bar plot in another web page.


Calling these services in your code

In a shell script, it's probably easiest to use 'curl' to call the URL - for example:

$ JOBID=${LOGNAME}:`date -Ins`

$ curl "http://ws.mwatelescope.org/progress/create?jobid=${JOBID}&desc=My+Job"

$ echo "View progress of this job at https://ws.mwatelescope.org/progress/show?jobid=${JOBID}"

That sets the JOBID variable to something unique (login name, followed by an ISO timestamp), displays the viewing URL to the user, and calls the create service. You would then pass the JOBID variable to the worker processes.

Note that if you have an spaces in your description, you need to turn them into '+' characters in the URL passed to curl, or use the --data-urlencode option in the curl command, eg:

$ curl -G --data-urlencode jobid="${JOBID}" --data-urlencode desc="This is my job" http://ws.mwatelescope.org/progress/create

If you're calling these services from Python, the easiest way is to use the requests library - for example:

>>> import requests

>>> requests.get('https://ws.mwatelescope.org/progress/update', params={'jobid':sys.argv[1], 'workid':'flagger1', 'current':7, 'total':50, 'desc':'lots of RFI')