Getting started

In this document we show how to build and run neuroimaging pipelines using Nipype and pypes.

Full example

import os

from hansel import Crumb
from neuro_pypes.run import run_debug
from neuro_pypes.config import update_config
from neuro_pypes.io import build_crumb_workflow
from neuro_pypes.anat import attach_spm_anat_preprocessing
from neuro_pypes.fmri import attach_rest_preprocessing

# root path to my data
base_dir = "/home/pyper/data/cobre/raw"

# the configuration file path
config_file = "/home/pyper/data/cobre_config.yml"

# define the Crumb filetree of my image database
data_path = os.path.join(base_dir, "{subject_id}", "session_1", "{modality}", "{image}")

# create the filetree Crumb object
data_crumb = Crumb(data_path, ignore_list=[".*"])

# the different worflows that I will use with any given name
attach_functions = {"spm_anat_preproc": attach_spm_anat_preprocessing,
                    "spm_rest_preproc": attach_rest_preprocessing,}

# the specific parts of the `data_crumb` that define a given modality.
# **Note**: the key values of this `crumb_arguments` must be the same as expected
# in the functions in `attach_functions`.
crumb_arguments = {'anat': [('modality', 'anat_1'),
                            ('image',    'mprage.nii.gz')],
                   'rest': [('modality', 'rest_1'),
                            ('image',    'rest.nii.gz')],
                  }

# the pypes configuration file with workflow parameter values
update_config(config_file_path)

# output and working directory paths
output_dir = os.path.join(os.path.dirname(base_dir), "out")
cache_dir  = os.path.join(os.path.dirname(base_dir), "wd")

# build the workflow
wf = build_crumb_workflow(attach_funcs,
                          data_crumb=data_crumb,
                          in_out_kwargs=crumb_arguments,
                          output_dir=output_dir,
                          cache_dir=cache_dir,)

# plot the workflow and then run it
run_wf(wf, plugin="MultiProc", n_cpus=4)

This example is explained step-by-step in the following sections.

Organize the data files

First, you should organize your dataset. This guide explains good ways to organize it. You can also have a look at the BIDS project.

Unless you have different necessities, we recommend using the following tree structure: {base_dir}/raw/{subject_id}/{session_id}/{image_files}.

Declaring the data

Now let's start programming. The first step is to declare and explain the file tree for the pipeline input.

Some imports first:

import os

from hansel import Crumb

Let's imagine the base folder where our data is:

base_dir = "/home/pyper/data/cobre/raw"

Now we specify the tree data structure using hansel.Crumb. We use os.path.join to build the path until the input files of the pipeline. Each part (subfolder) in this path that has many values, you enclose it with curly braces and put a sensible name for it. For example, in the level where all the folder named as subject identifiers, I will put "{subject_id}".

data_path = os.path.join(base_dir, "{subject_id}", "session_1", "{modality}", "{image}")

data_crumb = Crumb(data_path, ignore_list=[".*"])
#

Now in this way we have the data structure tree defined.

print(data_crumb)
>>> Crumb("/home/pyper/data/cobre/raw/{subject_id}/session_1/{modality}/{image}")

Choosing the pipelines

Once the data is defined, we have to specify the pipelines we want to run against these data. Let's imagine we want to run the anatomical and the function pre-processing pipelines.

First we import the attach functions to build those pipelines:

from neuro_pypes.anat import attach_spm_anat_preprocessing
from neuro_pypes.fmri import attach_rest_preprocessing

Then we specify and give names to the pipelines:

attach_functions = {"spm_anat_preproc": attach_spm_anat_preprocessing,
                    "spm_rest_preproc": attach_rest_preprocessing,}

One last thing, we have to link each file type to the corresponding pipeline. The attach_spm_anat_preprocessing expects an anat input and the attach_rest_preprocessing expects a rest input. So we have to specify the parameters in the data tree for each of these (check the argument names we used before):

crumb_arguments = {'anat': [('modality', 'anat_1'),
                            ('image',    'mprage.nii.gz')],
                   'rest': [('modality', 'rest_1'),
                            ('image',    'rest.nii.gz')],
                  }

Configuration

From pypes you can set up the internal parameters of the pipelines.

You can either have a separate configuration file (in JSON, YAML, .ini, etc...), dealt by Kaptan.

You can also set configuration parameters with a Python dict.

Pypes provides a function named update_config where you can input a dict or a file path to update the global configuration state before building the pipelines.

We recommend YAML: have a look at the example config file.

from neuro_pypes.config import update_config

if config_file_path:
    update_config(config_file_path)

if params_dict:
    update_config(params_dict)

See section Configuration for more details on how this works.

Build

Finally we define the output and working folders:

output_dir = path.join(path.dirname(base_dir), "out")

cache_dir = path.join(path.dirname(base_dir), "wd")

And we instantiate the workflow with the variables we created before:

from neuro_pypes.io import build_crumb_workflow

wf = build_crumb_workflow(attach_funcs,
                          data_crumb=data_crumb,
                          in_out_kwargs=crumb_arguments,
                          output_dir=output_dir,
                          cache_dir=cache_dir,)

Run

To run the workflow we can use two functions, with or without debugging. The one in debug mode will call a debugger if any exception is raised in the end. These functions are: neuro_pypes.run.run_debug and neuro_pypes.run.run_wf. They have the same arguments as the workflow wf.run() function.

from neuro_pypes.run import run_debug

run_debug(wf, plugin="MultiProc", n_cpus=4)

Great, you are done!

Now, check the first part of the Nipype console output, and if everything went fine go for a coffee, and wait for your results.