Contents
Introduction
Ok, things start to become clearer, but this product is called evQueue, where are these queues? For the moment I've seen none!
Well, in fact you already have used them, but just with the default parameters. evQueue ships with a default queue (well named 'default'). If you take one of your previous workflows and go to the 'Queue & retry' tab of one task, you will see that it is assigned to this default queue.
Queues are used to handle parallelism (if set to a concurrency above 1) or mutual exclusion (if set to a concurrency of 1). These two concepts are very useful for different things:
- Parallelism
It is used to speed up CPU intensive tasks, like images resizing or compression. Generally, you will use a loop to instanciate a bunch of tasks that will run in parallel. In this case you'll adapt the concurrency depending on your number of CPU. - Mutual exclusion
You might have tasks that use shared ressources that should not be accessed in parallel. In this case you'll use a queue with a concurrency of 1. This means that all tasks of all workflows attached to this queue will never run in parallel. This can be used to modify databases of filesystems.
The example workflow on photo resize (see loops) already uses parallel task execution for the convert script. However, it might not be slow enough for you to see it. So we will simulate a CPU intensive task with a sleep, for demonstration purposes.
You can download the resulting workflow here : loopsleep.xml.
Create the queue
Before we start, we need to create a new queue. Go to the Settings -> Queues menu and add a new queue ( icon).
Name it sleep and set a concurrency of 2. The scheduler can be left to 'default'.
Create the workflow
Create a new workflow, name it loopsleep.
As we said, in most cases parallel tasks are instanciated with loops. So we need something to loop on. Here, we'll use the following shell script:
#!/bin/bash
echo "<root>";
for((i=0;i<10;i++))
do
echo "<item />";
done
echo "</root>";
Create a "script" type task named 'loop' and copy/paste the above script. Go to "Output" tab and set "Output method" to XML.
Add a new jop (under 'loop') and set its path to '/bin/slep 5'. Go to the 'Queue & retry' tab and choose the 'sleep' queue.
Edit the job containing the sleep task and go to the 'Conditions & loops' tab. Open the helper ( icon) next to the loop input. Set it like this:
You now have the final workflow:
Understand what is happening
The next step is to launch our brand new workflow. It will run for exactly 25s: 2 tasks run simultaneously for 5 seconds, then 2 again... until we reach the 10 instanciated tasks.
There are several things to note here. First, when the instance is running you can see the icon in its status, indicating that some tasks are queued. This means that these tasks have not yet been launched due to concurrency limit on their queue. They will be launched as soon as a running task ends.
If you have a look at the instance detail (while running) you can see that some tasks are already terminated, more specifically 2 tasks are running and the others are queued:
One more last interesting thing to look is the System state -> Queues screen. You should see something like this:
