Pipelines¶
The Processes tab allows you to submit tasks one at a time. This is great if you are exploring different Apps or experimenting with different configurations that can best process your datasets. However, once you find the optimal set of Apps, you would probably want to run it across many subjects and submitting them one by one is simply not a good option.
Brainlife allows you to setup a series of submission rules called pipeline rules. Instead of describing the entire workflow that you submit once (or re-submit if something fails), you will define a set of individual rules which will be continuously evaluated until you deactivate them. It is similar to how a factory assembly line produces products. When a subject fails to produce an output dataset for a specific rule, you can examine and handle it manually. Once you are able to produce a valid output, the rest of the pipeline rules will pick it up as if it came from the original rule.
Setting up Pipeline Rule¶
To setup a new pipeline rule, go to Project > Pipelines tab and click a plus button at the bottom right corner of the page.
Each rule will be responsible for submitting a specific App with a specific set of configuration. Enter Name field, and search for the App that you'd like to submit. Once you select an App, you will be able to set its configuration parameters.
All Brainlife Apps have a defined list of input datatypes that Apps needs to run. Using this information, Brainlife will look for any subject that provides all input datatypes required by the App, and submit new process for each subject found to run your App. If you have more than one dataset that matches the required datatype for a subject, you can specify which datasets to use by specifying a dataset tags (not datatype tags). By default, it will use the latest dataset available for a given datatype.
When you are submitting your first rule, you probably don't have any dataset archived inside your project. If you'd like to use datasets from other project, you can specify the Project field to look for the input datasets there.
Above rule will submit processes for each subject found on ABIDE2 project that provides dwi datatype with a dataset tag of "ABIDEII-BNI_1".
Brainlife will only submit new process if it hasn't submitted a new process for each subject. Brainlife also won't submit new process if your project already has an output datasets (maybe generated by other rules, or generated manually). To be more specific about which datasets are generated by which rule, you can specify output dataset tags under outputs section.
You can leave this default if you know you there won't be any other App generating the same output datatype. We recommend to always set output dataset tags just in case.
Lastly, you can set a Subject Filtering which limits the subjects that gets processed.
Above example will make this rule to only submit for subjects with names that start with "100" or "200". When you are setting up your first rule, it's always good to limit number of subjects to make sure your rule is setup correctly.
Hint
There are number of regular expression tutorials available online. Also, please feel free to send us your question via Brainlife slack team.
Monitoring Pipeline Rules¶
Once you submit your pipeline rule, it should start submitting processes and you can monitor them under the processes tab.
You can treat these processes as you normally do with any processes that you normally submit manually; examine outputs, stop, restart, etc.. The output datasets will be automatically archived once each task have completed successfully.
Note
If you remove a process or task, Brainlife will resubmit another process to handle that subject if the subject has all required input datasets and has not produce the output from the requested app yet. If you don't want them to be resubmitted, please remove or deactivate your rule.
Troubleshooting Pipeline Rules¶
Once you submit your pipeline rule, you can monitor the status of the pipeline under the Log section
Information here should help you troubleshoot what Brainlife is doing with your rule, and most importantly, why it's not submitting proceses.