MRBrainS challenge online evaluation framework for brainimage segmentation in 3T MRI scansCitation for published version (APA)Mendrik A M Vincken K L Kuijf H J Breeuwer M Bouvy W H de Bresser J Alansary A de BruijneM Carass A El-Baz A Jog A Katyal R Khan A R van der Lijn F Mahmood Q Mukherjee R vanOpbroek A Paneri S Pereira S Viergever M A (2015) MRBrainS challenge online evaluationframework for brain image segmentation in 3T MRI scans Computational Intelligence and Neuroscience 20151-16 [813696] httpsdoiorg1011552015813696

DOI1011552015813696

Document status and datePublished 01012015

Document VersionPublisherrsquos PDF also known as Version of Record (includes final page issue and volume numbers)

Please check the document version of this publication

bull A submitted manuscript is the version of the article upon submission and before peer-review There can beimportant differences between the submitted version and the official published version of record Peopleinterested in the research are advised to contact the author for the final version of the publication or visit theDOI to the publishers websitebull The final author version and the galley proof are versions of the publication after peer reviewbull The final published version features the final layout of the paper including the volume issue and pagenumbersLink to publication

General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors andor other copyright ownersand it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights

bull Users may download and print one copy of any publication from the public portal for the purpose of private study or research bull You may not further distribute the material or use it for any profit-making activity or commercial gain bull You may freely distribute the URL identifying the publication in the public portal

If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act indicated by the ldquoTavernerdquo license above pleasefollow below link for the End User Agreementwwwtuenltaverne

Take down policyIf you believe that this document breaches copyright please contact us atopenaccesstuenlproviding details and we will investigate your claim

Download date 04 Feb 2021

Research ArticleMRBrainS Challenge Online Evaluation Framework forBrain Image Segmentation in 3T MRI Scans

Adrieumlnne M Mendrik1 Koen L Vincken1 Hugo J Kuijf1 Marcel Breeuwer23

Willem H Bouvy4 Jeroen de Bresser5 Amir Alansary6 Marleen de Bruijne78

Aaron Carass9 Ayman El-Baz6 Amod Jog9 Ranveer Katyal10 Ali R Khan1112

Fedde van der Lijn7 Qaiser Mahmood13 Ryan Mukherjee14 Annegreet van Opbroek7

Sahil Paneri10 Seacutergio Pereira15 Mikael Persson13 Martin Rajchl1116 Duygu Sarikaya17

Oumlrjan Smedby1819 Carlos A Silva15 Henri A Vrooman7 Saurabh Vyas14

Chunliang Wang1819 Liang Zhao17 Geert Jan Biessels4 and Max A Viergever1

1 Image Sciences Institute University Medical Center Utrecht 3584 CX Utrecht Netherlands2 Philips Healthcare 5680 DA Best Netherlands3 Faculty of Biomedical Engineering Eindhoven University of Technology 5600 MB Eindhoven Netherlands4 Department of Neurology Brain Center Rudolf Magnus University Medical Center Utrecht 3584 CX Utrecht Netherlands5 Department of Radiology University Medical Center Utrecht 3584 CX Utrecht Netherlands6 BioImaging Laboratory Bioengineering Department University of Louisville Louisville KY 40292 USA7 Biomedical Imaging Group Rotterdam Departments of Medical Informatics and Radiology Erasmus MC3015 CN Rotterdam Netherlands

8 Department of Computer Science University of Copenhagen 2100 Copenhagen Denmark9 Image Analysis and Communications Laboratory Department of Electrical and Computer EngineeringJohns Hopkins University Baltimore MD 21218 USA

10Department of Electronics and Communication Engineering The LNM Institute of Information Technology Jaipur 302031 India11 Imaging Laboratories Robarts Research Institute London ON Canada N6A 5B712Department of Medical Biophysics Western University London ON Canada N6A 3K713Signals and Systems Chalmers University of Technology 41296 Gothenburg Sweden14Applied Physics Laboratory Johns Hopkins University Laurel MD 20723 USA15Department of Electronics University of Minho 4800-058 Guimaraes Portugal16Department of Computing Imperial College London London SW7 2AZ UK17Computer Science and Engineering Department State University of New York at Buffalo Buffalo NY 14260-2500 USA18Center for Medical Imaging Science and Visualization Linkoping University 58185 Linkoping Sweden19Department of Radiology and Department of Medical and Health Sciences Linkoping University 58185 Linkoping Sweden

Correspondence should be addressed to Adrienne M Mendrik ammendrikgmailcom

Received 10 July 2015 Accepted 19 August 2015

Academic Editor Jussi Tohka

Copyright copy 2015 Adrienne M Mendrik et al This is an open access article distributed under the Creative Commons AttributionLicense which permits unrestricted use distribution and reproduction in any medium provided the original work is properlycited

Many methods have been proposed for tissue segmentation in brain MRI scans The multitude of methods proposed complicatesthe choice of one method above others We have therefore established the MRBrainS online evaluation framework for evaluating(semi)automatic algorithms that segment gray matter (GM) white matter (WM) and cerebrospinal fluid (CSF) on 3T brain MRIscans of elderly subjects (65ndash80 y) Participants apply their algorithms to the provided data after which their results are evaluatedand ranked Full manual segmentations of GM WM and CSF are available for all scans and used as the reference standard Fivedatasets are provided for training and fifteen for testing The evaluated methods are ranked based on their overall performance to

Hindawi Publishing CorporationComputational Intelligence and NeuroscienceVolume 2015 Article ID 813696 16 pageshttpdxdoiorg1011552015813696

2 Computational Intelligence and Neuroscience

segment GM WM and CSF and evaluated using three evaluation metrics (Dice H95 and AVD) and the results are published onthe MRBrainS13 website We present the results of eleven segmentation algorithms that participated in the MRBrainS13 challengeworkshop at MICCAI where the framework was launched and three commonly used freeware packages FreeSurfer FSL andSPMTheMRBrainS evaluation framework provides an objective and direct comparison of all evaluated algorithms and can aid inselecting the best performing method for the segmentation goal at hand

1 Introduction

Multiple large population studies [1ndash3] have shown theimportance of quantifying brain structure volume forexample to detect or predict small vessel disease andAlzheimerrsquos disease In clinical practice brain volumetry canbe of value in disease diagnosis progression and treatmentmonitoring of a wide range of neurologic conditionssuch as Alzheimerrsquos disease dementia focal epilepsyParkinsonism and multiple sclerosis [4] Automatic brainstructure segmentation in MRI dates back to 1985 [5] andmany methods have been proposed since then Howeverthe multitude of methods proposed [6ndash13] complicatesthe choice for a certain method above others As earlyas 1986 Price [14] stressed the importance of comparingdifferent approaches to the same type of problem Variousstudies have addressed this issue and evaluated differentbrain structure segmentation methods [15ndash19] Howeverseveral factors complicate direct comparison of differentapproaches Not all algorithms are publicly available andif they are researchers who use them are generally not asexperienced with these algorithms as they are with their ownalgorithm in terms of parameter tuning which could resultin a bias towards their own method This problem does notexist when researchers apply their own method to publiclyavailable data Therefore publicly available databases likethe ldquoAlzheimerrsquos Disease Neuroimaging Initiativerdquo (ADNI)(httpadniloniuscedu) the ldquoInternet Brain SegmentationRepositoryrdquo (IBSR) (httpwwwnitrcorgprojectsibsr) theCANDI Share Schizophrenia Bulletin 2008 (httpswwwnitrcorgprojectscs schizbull08) [20] and Mindboggle(httpwwwmindboggleinfo) [21] are important initiativesto enable comparison of various methods on the same dataHowever due to the use of subsets of the available data anddifferent evaluation measures direct comparison can be pro-blematic To address this issue grand challenges in bio-medical image analysis were introduced in 2007 [22]Participants in these competitions can apply their algorithmsto the provided data after which their results are evaluatedand ranked by the organizersMany challenges (httpgrand-challengeorgAll Challenges) have been organized sincethen providing an insight into the performance of automaticalgorithms for specific tasks in medical image analysis

In this paper we introduce the MRBrainS challenge eval-uation framework (httpmrbrains13isiuunl) an onlineframework to evaluate automatic and semiautomatic algo-rithms that segment gray matter (GM) white matter (WM)and cerebrospinal fluid (CSF) in 3T brain MRI scans ofolder (mean age 71) subjects with varying degrees of atrophyand white matter lesions This framework has three mainadvantages Firstly researchers apply their own segmentationalgorithms to the provided data Parameters are optimally

tuned to achieve the best possible performance Secondlyall algorithms are applied to the exact same data and thereference standard of the test data is unknown to the par-ticipating researchers Thirdly the evaluation algorithm andmeasures are the same for all evaluated algorithms enablingdirect comparison of the various algorithms The frameworkwas launched at the MRBrainS13 challenge workshop at theMedical Image Computing and Computer Assisted Interven-tion (MICCAI) conference on September 26th in 2013 Eleventeams participated in the challenge workshop with a widevariety of segmentation algorithms the results for which arepresented in this paper and provide a benchmark for theproposed evaluation framework In addition we evaluatedthree commonly used freeware packages on the evaluationframework FreeSurfer (httpsurfernmrmghharvardedu)[23 24] FSL (httpfslfmriboxacukfslfslwiki) [25] andSPM (httpwwwfilionuclacukspm) [26]

2 Materials and Methods

21 Evaluation Framework TheMRBrainS evaluation frame-work is set up as follows Multisequence (T1-weighted T1-weighted inversion recovery and T2-weighted fluid atten-uated inversion recovery) 3T MRI scans of twenty sub-jects are available for download on the MRBrainS website(httpmrbrains13isiuunl) The data is described in moredetail in Section 211 All scans were manually segmentedinto GM WM and CSF These manual segmentations areused as the reference standard for the evaluation frameworkThe annotation process for obtaining the reference standardis described in Section 212 For five of the twenty datasetsthe reference standard is provided on the website and can beused for training an automatic segmentation algorithm Theremaining fifteen MRI datasets have to be segmented by theparticipating algorithms into GM WM and CSF For thesefifteen datasets the reference standard is not provided onlineThe segmentation results can be submitted on the MRBrainSwebsite With each submission a short description of thesegmentation algorithm has to be provided which shouldat least describe the algorithm the used MRI sequenceswhether the algorithm is semi- or fully automatic and theaverage runtime of the algorithm The segmentation resultsare then evaluated (Section 213) and ranked (Section 214)by the organizers and the results are presented on the websiteMore information on how to use the evaluation frameworkis provided in the details section of the MRBrainS website(httpmrbrains13isiuunldetailsphp)

211 Data The focus was on brain segmentation in thecontext of ageing Twenty subjects (mean age plusmn SD = 71 plusmn 4years 10 male 10 female) were selected from an ongoing

Computational Intelligence and Neuroscience 3

cohort study of older (65ndash80 years of age) functionallyindependent individuals without a history of invalidatingstroke or other brain diseases [27] This study was approvedby the local ethics committee of the University MedicalCenter Utrecht (Netherlands) and all participants signed aninformed consent form To be able to test the robustnessof the segmentation algorithms in the context of ageing-related pathology the subjects were selected to have varyingdegrees of atrophy and white matter lesions Scans withmajor artefacts were excluded MRI scans were acquired ona 30 T Philips AchievaMR scanner at the University MedicalCenter Utrecht (Netherlands) The following sequences wereacquired and used for the evaluation framework 3D T1 (TR79ms TE 45ms) T1-IR (TR 4416ms TE 15ms and TI400ms) and T2- FLAIR (TR 11000ms TE 125ms and TI2800ms) Since the focus of the MRBrainS evaluation frame-work is on comparing different segmentation algorithms weperformed two preprocessing steps to limit the influenceof different registration and bias correction algorithms onthe segmentation results The sequences were aligned byrigid registration using Elastix [28] and bias correction wasperformed using SPM8 [29] After registration the voxel sizewithin all provided sequences (T1 T1 IR and T2 FLAIR)was 096 times 096 times 300mm3 The original 3D T1 sequence(voxel size 10 times 10 times 10mm3) was provided as well Fivedatasets that were representative for the overall data (2 male3 female varying degrees of atrophy andwhitematter lesions)were selected for training The remaining fifteen datasets areprovided as test data

212 Reference Standard Manual segmentations were per-formed to obtain a reference standard for the evaluationframework All axial slices of the 20 datasets (096 times 096times 300mm3) were manually segmented by trained researchassistants in a darkened room with optimal viewing con-ditions All segmentations were checked and corrected bythree experts a neurologist in training a neuroradiologistin training and a medical image processing scientist Toperform the manual segmentations an in-house developedtool based on MeVisLab (MeVis Medical Solutions AGBremen Germany) was used employing a freehand splinedrawing technique [30] The closed freehand spline drawingtechnique was used to delineate the outline of each brainstructure starting at the innermost structures (Figure 1(a))working outward The closed contours were converted tohard segmentations and the inner structures were itera-tively subtracted from the outer structures to construct thefinal hard segmentation image (Figure 1(b)) The followingstructures were segmented and are available for trainingcortical gray matter (1) basal ganglia (2) white matter (3)white matter lesions (4) peripheral cerebrospinal fluid (5)lateral ventricles (6) cerebellum (7) and brainstem (8)Thesestructures can be merged into gray matter (1 2) white matter(3 4) and cerebrospinal fluid (5 6) The cerebellum andbrainstem are excluded from the evaluation All structureswere segmented on theT1-weighted scans thatwere registeredto the FLAIR scans except for the white matter lesions(WMLs) and the CSF outer border (used to determine

the intracranial volume) The WMLs were segmented onthe FLAIR scan by the neurologist in training and checkedand corrected by the neuroradiologist in training The CSFouter border was segmented using both the T1-weightedand the T1-weighted IR scan since the T1-weighted IR scanshows higher contrast at the borders of the intracranialvolumeTheCSF segmentation includes all vessels (includingthe superior sagittal sinus and the transverse sinuses) andnonbrain structures such as the cerebral falx and choroidplexuses

213 Evaluation To evaluate the segmentation results weuse three types of measures a spatial overlap measure aboundary distance measure and a volumetric measure TheDice [31] coefficient is used to determine the spatial overlapand is defined as

119863 =2 |119860 cap 119866|

|119860| + |119866|

sdot 100 (1)

where 119860 is the segmentation result 119866 is the referencestandard and 119863 is the Dice expressed as percentages The95th-percentile of theHausdorffdistance is used to determinethe distance between the segmentation boundaries Theconventional Hausdorff distance uses the maximum whichis very sensitive to outliers To correct for outliers we usethe 95th-percentile of theHausdorff distance by selecting the119870th ranked distance as proposed by Huttenlocher et al [32]

ℎ95 (119860 119866) =

95119870

th119886isin119860

min119892isin119866

1003817100381710038171003817119892 minus 1198861003817100381710038171003817 (2)

where 95119870th119886isin119860

is the119870th rankedminimumEuclidean distancewith 119870119873

119886= 95 119860 is the set of boundary points

1198861 119886

119873119886

of the segmentation result and 119866 is the set ofboundary points 119892

1 119892

119873119892

of the reference standard The95th-percentile of the Hausdorff distance is defined as

11986795(119860 119866) = max (ℎ

95(119860 119866) ℎ

95(119866 119860)) (3)

The third measure is the percentage absolute volume differ-ence defined as

AVD =10038161003816100381610038161003816119881119886minus 119881119892

10038161003816100381610038161003816

119881119892

sdot 100 (4)

where 119881119886is the volume of the segmentation result and 119881

119892

is the volume of the reference standard These measures areused to evaluate the following brain structures in each of thefifteen test datasets GM WM CSF brain (GM + WM) andintracranial volume (GM + WM + CSF) The brainstem andcerebellum are excluded from the evaluation

214 Ranking To compare the segmentation algorithmsthat participate in the MRBrainS evaluation framework thealgorithms are ranked based on their overall performance tosegment GMWM and CSF Each of these components (119862 =GMWMCSF) is evaluated by using the three evaluationmeasures (119872 = 119863119867

95AVD) described in Section 213

4 Computational Intelligence and Neuroscience

(a) (b) (c) (d) (e)

Figure 1 Example of themanually drawn contours (a) the resulting hard segmentationmap (GM light blueWM yellow andCSF dark blue)that is used as the reference standard (b) the T1-weighted scan (c) the T1-weighted inversion recovery (IR) scan (d) and the T2-weightedfluid attenuated inversion recovery (FLAIR) scan (e)

For each component 119888 isin 119862 and each evaluation measure119898 isin119872 the mean and standard deviation are determined over all15 test datasets The segmentation algorithms are then sortedon the mean 119863 value in descending order and on the mean11986795

and AVD value in ascending order Each segmentationalgorithm receives a rank (119903) between 1 (ranked best) and 119899(number of participating algorithms) for each component 119888and each evaluation measure 119898 The final ranking is basedon the overall score of each algorithm which is the sum overall ranks defined as

119904 =

|119872|

sum

119898=0

|119862|

sum

119888=0

119903119898119888 (5)

where 119903119898119888

is the rank of the segmentation algorithm formeasure 119898 of component 119888 For the final ranking 119903 theoverall scores 119904 are sorted in ascending order and rankedfrom 1 to 119899 In case two or more algorithms have equalscores the standard deviation over all 15 test datasets is takeninto account to determine the final rank The segmentationalgorithms are then sorted on the standard deviation inascending order and ranked for each component 119888 and eachevaluation measure119898 The overall score is determined using(5) and the algorithms are sorted based on this score inascending order and ranked from 1 to 119899 The algorithms thathave equal overall scores based on the mean value are thenranked based on this standard deviation rank

22 EvaluatedMethods Theevaluation framework describedin Section 21 was launched at the MRBrainS13 challengeworkshop at the Medical Image Computing and ComputerAssisted Intervention (MICCAI) conference on September26th in 2013 For the workshop challenge the test datasetswere split into twelve off-site and three on-site test datasetsFor the off-site part teams could register on the MRBrainSwebsite (httpmrbrains13isiuunl) and download the fivetraining and twelve test datasets A time slot of eight weekswas available for teams to download the data train theiralgorithms segment the test datasets and submit their resultson the website Fifty-eight teams downloaded the data ofwhich twelve submitted their segmentation results The eval-uation results were reported to the twelve teams and all teamssubmitted a workshop paper to the MRBrainS13 challengeworkshop at MICCAI Eleven teams presented their results

at the workshop and segmented the three on-site test datasetslive at the workshop within a time slot of 35 hours Thesealgorithms provide a benchmark for the proposed evaluationframework and are briefly described in Sections 221ndash2211in alphabetical order of teamsrsquo names The teamsrsquo namesare used in the paper to identify the methods For a fulldescription of the methods we refer to the workshop papers[33ndash43] In Section 2212 we describe the evaluated freewarepackages

221 BIGR2 [37] This multifeature SVM [37] method clas-sifies voxels by using a Support Vector Machine (SVM)classifier [44] with a Gaussian kernel Besides spatial featuresand intensity information from all three MRI sequences theSVM classifier incorporates Gaussian-scale-space features tofacilitate a smooth segmentation Skull stripping is performedby nonrigid registration of the masks of the training imagesto the target image

222 Bigr neuro [41] This auto-kNN [45] method is basedon an automatically trained kNN-classifier First a proba-bilistic tissue atlas is generated by nonrigidly registering themanually annotated atlases to the subject of interest Trainingsamples are obtained by thresholding the probabilistic atlasand subsequently pruning the feature space White matterlesions are detected by applying an adaptive threshold deter-mined from the tissue segmentation to the FLAIR sequence

223 CMIV [43] A statistical-model-guided level-setmethod is used to segment the skull brain ventricles andbasal ganglia Then a skeleton-based model is created byextracting the midsurface of the gray matter and definingthe thickness This model is incorporated into a level-setframework to guide the cortical gray matter segmentationThe coherent propagation algorithm [46] is used to acceleratethe level-set evolution

224 Jedi MindMeld [42] Thismethod starts by preprocess-ing the data via anisotropic diffusion For each 2D slice of alabeled dataset the canny edge pixels are extracted and theTourist Walk is computed This is done for axial sagittal andcoronal views Machine learning is used with these featuresto automatically label edge pixels in an unlabeled dataset

Computational Intelligence and Neuroscience 5

Finally these labels are used by the Random Walker forautomatic segmentation

225 LNMBrains [35] The voxel intensities of all MRIsequences are modelled as a Gaussian distribution for eachlabel The parameters of the Gaussian distributions areevaluated asmaximum likelihood estimates and the posteriorprobability of each label is determined by using Bayesianestimation A feature set consisting of regional intensitytexture spatial location of voxels and the posterior prob-ability estimates is used to classify each voxel into CSFWM GM or background by using a multicategory SVMclassifier

226 MNAB [38] This method uses Random DecisionForests to classify the voxels into GM WM and CSF Itstarts by a skull stripping procedure followed by an intensitynormalization of each MRI sequence Feature extraction isthen performed on the intensities posterior probabilitiesneighborhood statistics tissue atlases and gradient mag-nitude After classification isolated voxels are removed bypostprocessing

227 Narsil [34] This is a model-free algorithm that usesensembles of decision trees [47] to learn the mapping fromimage features to the corresponding tissue label The ensem-bles of decision trees are constructed from correspondingimage patches of the provided T1 and FLAIR scans withmanual segmentations The N3 algorithm [48] was used foradditional inhomogeneity correction and SPECTRE [49] wasused for skull stripping

228 Robarts [39] Multiatlas registration [50] with the T1training images was used to propagate labels to generatesample histograms in a log-likelihood intensity model andprobabilistic shape priors These were employed in a MAPdata term and regularized via computation of a hierar-chical max-flow [51] A brain mask from registration ofthe T1-IR training images was used to obtain the finalresults

229 S2 QM [36] This method [52] is based on Bayesian-based adaptive mean shift and the voxel-weighted 119870-meansalgorithm The former is used to segment the brain into alarge number of clusters or modes The latter is employed toassign these clusters to one of the three components WMGM or CSF

2210 UB VPMLMed [40] This method creates a multiatlasby registering the training images to the subject imageand then propagating the corresponding labels to a fullyconnected graph on the subject image Label fusion thencombines the multiple labels into one label at each voxelwith intensity similarity based weighted voting Finally themethod clusters the graph using multiway cut in order toachieve the final segmentation

2211 UofL BioImaging [33] This is an automated MAP-based method aimed at unsupervised segmentation of differ-ent brain tissues from T1-weighted MRI It is based on theintegration of a probabilistic shape prior a first-order inten-sity model using a Linear Combination of Discrete Gaussians(LCDG) and a second-order appearance model These threefeatures are integrated into a two-level joint Markov-GibbsRandom Field (MGRF) model of T1-MR brain imagesSkull stripping was performed using BET2 [40] followed byan adaptive threshold-based technique to restore the outerborder of the CSF using both T1 andT1-IR this techniquewasnot described in [33] due to a US patent application [53] butis described in [54] This method was applied semiautomat-ically to the MRBrainS test data due to per scan parametertuning

2212 Freeware Packages Next to the methods evaluatedat the workshop we evaluated three commonly used free-ware packages for MR brain image segmentation FreeSurfer(httpsurfernmrmghharvardedu) [23 24] FSL (httpfslfmriboxacukfslfslwiki) [25] and SPM12 (httpwwwfilionuclacukspm) [26] All packages were applied usingthe default settings unless mentioned otherwise FreeSurfer(v530) was applied to the high resolution T1 sequence Themri label2vol tool was used to map the labels on the thickslice T1 that was used for the evaluation FSL (v50) wasdirectly applied to the thick slice T1 and provides both apveseg and a seg file as binary output We evaluated bothof these files The fractional intensity threshold parameterldquo119891rdquo of the BET tool that sets the brainnonbrain intensitythreshold was set according to [55] at 02 (Philips Achieva3T setting) SPM12 was directly applied to the thick sliceT1 sequence as well However it also provides the optionto add multiple MRI sequences Therefore we evaluatedSPM12 not only on the thick slice T1 sequence but addedthe T1-IR and the T2-FLAIR scan as well and tested variouscombinationsThe number of Gaussians was set according tothe SPM manual to two for GM two for WM and two forCSF

23 Statistical Analysis All evaluated methods werecompared to the reference standard In summary ofthe results the mean and standard deviation overall 15 test datasets were calculated per component(GM WM and CSF) and combination of components(brain intracranial volume) and per evaluation measure(Dice 95th-percentile Hausdorff distance and absolutevolume difference) for each of the evaluated methodsBoxplots were created using R version 303 (R projectfor statistical computing (httpwwwr-projectorg))Since white matter lesions should be segmented as whitematter the percentage of white matter lesion voxelssegmented as white matter (sensitivity) was calculatedfor each algorithm over all 15 test datasets to evaluatethe robustness of the segmentation algorithms againstpathology

6 Computational Intelligence and Neuroscience

3 Results

Table 1 presents the final ranking (119903) of the evaluatedmethodsthat participated in the workshop as well as the evalu-ated freeware packages During the workshop team UofLBioImaging ranked first and BIGR2 ranked second with onepoint difference in the overall score 119904 (5) However addingthe results of the freeware packages resulted in an equal scorefor UofL BioImaging and BIGR2 Therefore the standarddeviation rank was taken into account and BIGR2 is rankedfirst with standard deviation rank four and UofL BioImagingis ranked second with standard deviation rank eight Table 1further presents the mean standard deviation and rank foreach evaluation measure (119863119867

95 and AVD) and component

(GM WM and CSF) as well as the brain (WM + GM) andintracranial volume (WM+GM+CSF) Team BIGR2 scoredbest for theGMWM and brain segmentation and teamUofLBioImaging for the CSF segmentation Team Robarts scoredbest for the intracranial volume segmentation The boxplotsfor all evaluation measures and components are shown inFigures 2ndash4 and include the results of the freeware packagesFigure 5 shows an example of the segmentation results atthe height of the basal ganglia (slice 22 of test subject 9)The sensitivity of the algorithms to segment white matterlesions as WM and examples of the segmentation results inthe presence of white matter lesions (slice 31 of test subject3) are shown in Figure 6 Team UB VPML Med scores thehighest sensitivity of white matter lesions segmented as whitematter and is therefore most robust in the presence of thistype of pathology

4 Discussion

In this paper we proposed the MRBrainS challenge onlineevaluation framework to evaluate automatic and semiauto-matic algorithms for segmenting GM WM and CSF on3T multisequence (T1 T1-IR and T2-FLAIR) MRI scans ofthe brain We have evaluated and presented the results ofeleven segmentation algorithms that provide a benchmarkfor algorithms that will use the online evaluation frameworkto evaluate their performance Team UofL BioImaging andBIGR2 have equal overall scores but BIGR2 was rankedfirst based on the standard deviation ranking The evalu-ated methods represent a wide variety of algorithms thatinclude Markov random field models clustering approachesdeformable models and atlas-based approaches and classi-fiers (SVM KNN and decision trees) The presented evalua-tion framework provides an insight into the performance ofthese algorithms in terms of accuracy and robustness Variousfactors influence the choice for a certain method aboveothers We provide three measures that could aid in selectingthe method that is most appropriate for the segmentationgoal at hand a boundarymeasure (95th-percentile Hausdorffdistance 119867

95) an overlap measure (Dice coefficient 119863) and

a volume measure (absolute volume difference AVD) Allthree measures are taken into account for the final rankingof the methods This ranking was designed to get a quickinsight into how the methods perform in comparison to eachother The best overall method is the method that performs

well for all three measures and all three components (GMWM andCSF) However whichmethod to select depends onthe segmentation goal at hand Not all measures are relevantfor all segmentation goals For example if segmentation isused for brain volumetry [4] the overlap (119863) and volume(AVD) measures of the brain and intracranial volume (usedfor normalization [56]) segmentations are important to takeinto account On the other hand if segmentation is usedfor cortical thickness measurements the focus should be onthe gray matter boundary (119867

95) and overlap (119863) measures

Therefore the final ranking should be used to get a first insightinto the overall performance after which the performanceof the measures and components that are most relevant forthe segmentation goal at hand should be considered Besidesaccuracy robustness could also influence the choice for acertain method above others For example team UB VPMLMed shows a high sensitivity score for segmenting whitemat-ter lesions as white matter (Figure 6) and shows a consistentsegmentation performance of gray and white matter over all15 test datasets (Figures 2ndash4) This could be beneficial forsegmenting scans of populations with white matter lesionsbut is less important if the goal is to segment scans ofyoung healthy subjects In the latter case the most accuratesegmentation for gray and white matter (team BIGR2) ismore interesting If a segmentation algorithm is to be usedin clinical practice speed is an important consideration aswell The runtime of the evaluated methods is reported inTable 1 However these runtimes are merely an indicationof the required time since academic software is generallynot optimized for speed and the runtime is measured ondifferent computers andplatformsAnother relevant aspect ofthe evaluation framework is the comparison of multi- versussingle-sequence approaches For example most methodsstruggle with the segmentation of the intracranial volume onthe T1-weighted scan There is no contrast between the CSFand the skull and the contrast between the dura mater andthe CSF is not always sufficient Team Robarts used an atlas-based registration approach on the T1-IR scan (good contrastbetween skull and CSF) to segment the intracranial volumewhich resulted in the best performance for intracranialvolume segmentation (Table 1 Figures 2ndash4) Most methodsadd the T2-FLAIR scan to improve robustness against whitematter lesions (Table 1 Figure 6) Although using only theT1-weighted scan and incorporating prior shape information(team UofL BioImaging) can be very effective also thefreeware packages support this as well Since FreeSurfer isan atlas-based method it uses prior information and is themost robust of all freeware packages to white matter lesionsHowever adding the T2 FLAIR scan to SPM12 increasesrobustness against white matter lesions as well as comparedto applying SPM12 to the T1 scan only (Figure 6) In generalSPM12with the T1 and the T2-FLAIR sequence performswellin comparison to the other freeware packages (Table 1 andFigures 2ndash4) on the thick slice MRI scans Although addingthe T1-IR scan to SPM increases the performance of the CSFand ICV segmentations as compared to using only the T1 andT2-FLAIR sequence it decreases the performance of the GMand WM segmentations Therefore adding all sequences toSPM12 did not result in a better overall performance

Computational Intelligence and Neuroscience 7

Table 1 Results of the 11 evaluated algorithms presented at the workshop and the evaluated freeware packages on the 15 test datasets Thealgorithms are ranked (119903) based on their overall score (119904) by using (5) This score is based on the ranks of the gray matter (GM) white matter(WM) and cerebrospinal fluid (CSF) segmentation and the three evaluated measures Dice coefficient (119863 in ) 95th-percentile Hausdorffdistance (119867

95in mm) and the absolute volume difference (AVD in ) The rank 119903

119898119888denotes the rank based on the mean (120583) over all 15 test

datasets for each measure 119898 (0 119863 1 11986795 and 2 AVD) and component 119888 (0 GM 1 WM 2 CSF 3 brain (WM + GM) and 4 intracranial

volume (ICV = WM + GM + CSF)) Teams BIGR2 and UofL BioImaging and FreeSurfer and Jedi Mind Meld have equal scores based onthe mean (120583) therefore the ranking based on the standard deviation (120590) is taken into account to determine the final rank (BIGR2 120590 rank 4UofL BioImaging 120590 rank 8 FreeSurfer 120590 rank 13 and Jedi MindMeld 120590 rank 17) Columns 2 and 3 present the average runtime 119905 per scan inseconds (s) minutes (m) or hours (h) and the scans (T1 T1-weighted scan 3D T1 3D T1-weighted scan IR T1-weighted inversion recovery(IR) and F T1-weighted FLAIR) that are used for processing

119903 Team 119905 Scans

GM WM CSF

119904

Brain ICV119903001199031011990320119903011199031111990321119903021199031211990322

119903031199031311990323119903041199031411990324

119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590)

1 BIGR2 35m T1 IR F1 2 4 2 2 4 4 5 14

381 2 12 8 4 12

847 19 61 884 24 60 783 32 23 951 27 32 960 39 52(13) (04) (33) (12) (05) (51) (50) (08) (17) (05) (08) (16) (13) (11) (30)

2 UofLBioImaginglowast 6 s T1

5 1 9 4 1 13 2 2 138

2 1 12 5 2 5830 17 86 879 22 87 789 27 97 949 24 39 967 34 18(15) (03) (54) (20) (06) (66) (42) (05) (10) (06) (05) (20) (08) (06) (20)

3 CMIV 3m T1 F6 7 5 5 3 10 3 3 8

505 7 2 4 3 11

824 27 68 877 24 73 786 30 14 945 38 26 968 38 49(14) (04) (40) (16) (04) (38) (31) (04) (59) (05) (11) (22) (08) (13) (23)

4 UB VPMLMed 30m T1 IR F

4 4 2 1 5 7 8 13 1761

4 4 1 10 11 15833 21 59 886 27 71 748 43 31 946 28 24 948 66 77(13) (03) (53) (17) (04) (38) (71) (17) (19) (06) (04) (18) (20) (20) (41)

5 Bigr neuro 2 h T1 F7 13 3 6 6 9 6 4 10

647 10 10 7 5 8

815 37 59 873 30 73 782 32 16 940 46 36 963 39 35(17) (09) (42) (14) (04) (38) (47) (06) (14) (08) (14) (24) (12) (09) (27)

6 Robarts 16m 3D T1 IR11 3 15 8 8 6 1 1 13

6616 3 18 1 1 1

797 20 98 862 31 71 803 27 20 931 28 79 979 26 09(24) (01) (73) (13) (04) (62) (41) (05) (13) (16) (05) (36) (03) (04) (07)

7 Narsil 2m T1 F3 5 1 7 11 2 17 18 7

713 5 3 16 18 9

835 23 55 871 33 58 666 133 14 948 29 29 925 24 37(18) (04) (44) (13) (09) (53) (24) (54) (95) (05) (05) (20) (05) (89) (17)

8 SPM T1 F 3m T1 F8 9 16 9 7 1 9 14 2

759 14 14 6 14 4

812 29 10 860 30 52 741 46 10 939 58 53 966 82 15(22) (03) (85) (15) (01) (38) (34) (06) (47) (10) (22) (38) (02) (30) (10)

9 SPM T1 IR 3m T1 IR12 11 7 16 12 5 5 10 3

818 11 7 2 8 2

794 30 72 835 36 63 783 40 10 939 46 34 977 65 10(21) (04) (63) (21) (03) (46) (38) (06) (57) (08) (12) (28) (02) (13) (08)

10 MNAB 15m T1 IR F2 8 12 3 4 11 15 15 16

866 9 11 15 12 17

839 28 91 880 27 78 681 49 29 945 45 38 925 71 97(21) (09) (65) (12) (08) (40) (40) (22) (21) (10) (20) (32) (11) (42) (47)

11 SPM T1 3m T19 10 6 11 10 3 11 16 15

9110 8 6 9 13 13

803 30 69 856 31 60 707 53 23 939 44 32 953 81 55(24) (05) (68) (17) (01) (41) (38) (15) (157) (09) (16) (29) (09) (37) (37)

12 FSL Seg 10m T113 16 10 10 13 14 12 6 5

9913 13 4 12 6 6

787 43 86 860 37 115 699 34 12 933 55 30 942 53 34(22) (12) (63) (26) (08) (63) (28) (02) (103) (08) (14) (15) (08) (11) (15)

8 Computational Intelligence and Neuroscience

Table 1 Continued

119903 Team 119905 Scans

GM WM CSF

119904

Brain ICV119903001199031011990320119903011199031111990321119903021199031211990322

119903031199031311990323119903041199031411990324

119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590)

13 SPM T1 IR F 4m T1 IR F10 12 18 15 14 12 7 11 6

10512 15 13 3 15 3

801 30 139 836 38 84 769 41 12 936 59 51 977 82 12(24) (02) (96) (21) (05) (52) (31) (05) (60) (11) (18) (36) (02) (18) (09)

14 FSL PVSeg 10m T115 15 8 13 15 17 13 7 4

10711 16 9 13 7 7

777 43 84 848 38 197 695 34 11 936 61 35 942 53 34(26) (13) (65) (32) (09) (10) (22) (03) (58) (09) (16) (30) (08) (11) (15)

15 FreeSurfer 1 h 3D T116 6 17 12 9 8 18 12 18

11617 6 16 17 10 18

774 23 121 852 31 72 658 43 50 923 36 63 924 66 10(20) (06) (60) (22) (05) (46) (37) (06) (196) (08) (06) (34) (09) (04) (45)

16 Jedi MindMeld 27 s T1 IR F

14 14 11 17 16 15 10 8 11116

15 12 8 11 16 14778 39 89 806 45 137 733 37 18 932 54 35 943 20 68(59) (28) (76) (96) (36) (12) (35) (08) (90) (17) (48) (30) (17) (37) (31)

17 S2 QM 15 h T1 IR F17 17 13 14 17 18 16 9 9

13014 17 15 14 9 10

764 55 93 839 49 248 679 38 14 933 70 55 937 65 40(34) (30) (64) (34) (32) (10) (23) (06) (99) (14) (35) (45) (11) (14) (22)

18 LNMBrains 5m T1 IR F18 18 14 18 18 16 14 17 12

14518 18 17 18 17 16

728 68 95 783 68 153 688 77 20 885 86 74 892 204 79(53) (23) (76) (64) (35) (13) (66) (24) (13) (45) (28) (77) (48) (37) (83)

lowastSemiautomatic due to per scan parameter tuning

Besides the advantages of the MRBrainS evaluationframework there are some limitations that should be takeninto account The T1-weighted IR and the T2-weightedFLAIR scan were acquired with a lower resolution (096times 096 times 300mm3) than the 3D T1-weighted scan (10 times10 times 10mm3) To be able to provide a registered multise-quence dataset the 3D T1-weighted scan was registered tothe T2-weighted FLAIR scan and downsampled to 096 times096 times 300mm3 The reference standard is therefore onlyavailable for this resolution The decreased performance ofthe FreeSurfer GM segmentation as compared to the otherfreeware packages might be due to the fact that we evaluateon the thick slice T1 sequence instead of the high resolutionT1 Performing the manual segmentations to provide thereference standard is very laborsome and time consumingInstead of letting multiple observers manually segment theMRI datasets or letting one observer manually segmentthe MRI datasets twice much time and effort was spenton creating one reference standard that was as accurate aspossible Therefore we were not able to determine the inter-or intraobserver variability Finally we acknowledge that ourevaluation framework is limited to evaluating the accuracyand robustness over 15 datasets for segmenting GM WMand CSF on 3T MRI scans acquired on a Philips scanner ofa specific group of elderly subjects Many factors influencesegmentation algorithm performance such as the type ofscanner (vendor field strength) the acquisition protocolthe available MRI sequences and the type of subjects

Participating algorithmsmight have been designed for differ-ent types of MRI scans Therefore the five provided trainingdatasets are important for participants to be able to traintheir algorithms on the provided data Some algorithms aredesigned to segment only some components such as onlyGM and WM instead of all three components and usefreely available software such as the brain extraction tool[57] to segment the outer border of the CSF (intracranialvolume) We have chosen to base the final ranking on allthree components but it is therefore important to assess notonly the final ranking but the performance of the individualcomponents as well

Despite these limitations the MRBrainS evaluationframework provides an objective and direct comparison ofsegmentation algorithms The reference standard of the testdata is unknown to the participants the same evaluationmeasures are used for all evaluated algorithms and partici-pants apply their own algorithms to the provided data

In comparison to the online validation engine proposedby Shattuck et al [58] the MRBrainS evaluation frameworkuses 3T MRI data instead of 15T MRI data and evaluates notonly brain versus nonbrain segmentation but also segmenta-tion of gray matter white matter cerebrospinal fluid brainand intracranial volume The availability of many differenttypes of evaluation frameworks will aid in the developmentof more generic and robust algorithms For example inthe NEATBrainS (httpneatbrains15isiuunl) challengeresearchers were challenged to apply their algorithms to data

Computational Intelligence and Neuroscience 9

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

65

70

75

80

85

GM Dice

Dic

e (

)

(a)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

55

60

65

70

75

80

85

90

WM Dice

Dic

e (

)

(b)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

55

60

65

70

75

80

85CSF Dice

Dic

e (

)

(c)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

80

85

90

95

Brain (WM + GM) Dice

Dic

e (

)

(d)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

80

85

90

95

Dic

e (

)

Intracranial volume (WM + GM + CSF) Dice

(e)

Figure 2 Boxplots presenting the evaluation results for the Dice coefficient (1) of the gray matter (GM) white matter (WM) cerebrospinalfluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all 15 test datasets

10 Computational Intelligence and Neuroscience

2468

10121416

GM H-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(a)

2468

10121416

WMH-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(b)

5

10

15

20CSF H-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(c)

5

10

15

20H

-95

(mm

)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Brain (WM + GM) H-95

(d)

5

10

15

20

25

30

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Intracranial volume (WM + GM + CSF) H-95

(e)

Figure 3 Boxplots presenting the evaluation results for the 95th-percentile Hausdorff distance (3) of the gray matter (GM) white matter(WM) cerebrospinal fluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all15 test datasets

Computational Intelligence and Neuroscience 11

0

10

20

30

GM AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(a)

0

10

20

30

40WM AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(b)

0

20

40

60

80

CSF AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(c)

0

5

10

15

20

25

30AV

D (

)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Brain (WM + GM) AVD

(d)

0

5

10

15

20

25

30

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Intracranial volume (WM + GM + CSF) AVD

(e)

Figure 4 Boxplots presenting the evaluation results for the absolute volume difference (4) of the gray matter (GM) white matter (WM)cerebrospinal fluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all 15 testdatasets

12 Computational Intelligence and Neuroscience

T2

UofL

N

FS

S2

‐FLAIR

BioImaging

Narsil

L‐Seg

2_QM

T1‐IR

CMIV

SPM12_T1

SPM12_T1_

LNMBr

_FLAIR

_IR_FLAIR

ains

T1

UB VPML Me

SPM12_T1_ I

FSL‐PVSeg

R

d B

R

F

eference

igr_neuro

MNAB

reeSurfer

BIG

Rob

SPM1

Jedi Mi

R2

arts

2_T1

nd Meld

Figure 5 Illustration of the segmentation results at the height of the basal ganglia (test subject 9 slice 22) The basal ganglia should besegmented as gray matter The first three images show the three MRI sequences The fourth image (reference) is the manually segmentedreference standard (yellow white matter light blue gray matter and dark blue cerebrospinal fluid) The results of the participatingsegmentation algorithms are ordered from the best overall performance (BIGR2) to the worst overall performance (LNMBrains) Majordifferences are present in the segmentation of the basal ganglia and the outer border of the cerebrospinal fluid The arrows indicate examplelocations where the segmentation results differ from the ground truth

Computational Intelligence and Neuroscience 13

T2

UofL BioIm

Nars

FSL‐S

S2_Q

‐FLAIR

aging (860)

il (892)

eg (453)

M (565)

T1‐IR

CMIV (90

SPM12_T1_F

SPM12_T1_IR

LNMBrains

5) UB

(831) SP

_F (815) F

(286)

T1

VPML Med (9

M12_T1_IR (60

SL‐PVSeg (525

R

10) Bigr_

5) MN

) FreeS

eference

neuro (896)

AB (889)

urfer (846)

BIGR2

Robarts

SPM12_T

Jedi Mind M

(852)

(810)

1 (683)

eld (769)

Figure 6 Illustration of the segmentation results in the presence of whitematter lesions (test subject 3 slice 31)Whitematter lesions (WMLs)should be labeled asWM the sensitivity (percentage ofWML voxels is labeled asWM) over all 15 datasets is presented between brackets afterthe team nameThe first three images show the threeMRI sequences T2-weighted fluid attenuated inversion recovery T1-weighted inversionrecovery and T1-weighted scanThe fourth image (reference) is themanually segmented reference standard (red whitematter lesions yellowwhite matter light blue gray matter and dark blue cerebrospinal fluid)The results of the segmentation algorithms are ordered from the bestoverall performance (BIGR2) to the worst overall performance (LNMBrains)The arrows indicate example locations where the segmentationresults differ from the ground truth

14 Computational Intelligence and Neuroscience

fromboth theMRBrainS and theNeoBrainS [59] (brain tissuesegmentation in neonates) challenge Two methods [60ndash62]specifically designed for neonatal brain tissue segmentationshowed a high performance for tissue segmentation on theMRBrainS data Applying algorithms to different types of datahas the potential to lead to new insights and more robustalgorithms The MRBrainS evaluation framework remainsopen for new contributions At the time of writing 21teams had submitted their results on the MRBrainS website(httpmrbrains13isiuunlresultsphp)

5 Conclusion

The MRBrainS challenge online evaluation framework pro-vides direct and objective comparison of automatic andsemiautomatic methods to segment GM WM CSF brainand ICV on 3T multisequence MRI data The first elevenparticipating methods are evaluated and presented in thispaper as well as three commonly used freeware packages(FreeSurfer FSL and SPM12) They provide a benchmarkfor future contributions to the framework The final rankingprovides a quick insight into the overall performance of theevaluated methods in comparison to each other whereasthe individual evaluation measures (Dice 95th-percentileHausdorff distance and absolute volume difference) percomponent (GM WM CSF brain and ICV) can aid inselecting the best method for a specific segmentation goal

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was financially supported by IMDI Grant104002002 (Brainbox) fromZonMw the Netherlands Organ-isation for Health Research and Development within kindsponsoring by Philips theUniversityMedical CenterUtrechtand Eindhoven University of TechnologyThe authors wouldlike to acknowledge the following members of the UtrechtVascular Cognitive Impairment Study Group who were notincluded as coauthors of this paper but were involved in therecruitment of study participants and MRI acquisition at theUMC Utrecht (in alphabetical order by department) E vanden Berg M Brundel S Heringa and L J Kappelle of theDepartment of Neurology P R Luijten and W P Th MMali of the Department of Radiology and A Algra and GE H M Rutten of the Julius Center for Health Sciences andPrimary CareThe research of Geert Jan Biessels and the VCIgroupwas financially supported byVIDIGrant 91711384 fromZonMw and by Grant 2010T073 of the Netherlands HeartFoundation The research of Jeroen de Bresser is financiallysupported by a research talent fellowship of the Univer-sity Medical Center Utrecht (Netherlands) The research ofAnnegreet vanOpbroek andMarleen de Bruijne is financiallysupported by a research grant from NWO (the NetherlandsOrganisation for Scientific Research) The authors would

like to acknowledge MeVis Medical Solutions AG (BremenGermany) for providing MeVisLab Duygu Sarikaya andLiang Zhao acknowledge their Advisor Professor JasonCorsofor his guidance Duygu Sarikaya is supported by NIH 1 R21CA160825-01 and Liang Zhao is partially supported by theChina Scholarship Council (CSC)

References

[1] I Driscoll C Davatzikos Y An et al ldquoLongitudinal pattern ofregional brain volume change differentiates normal aging fromMCIrdquo Neurology vol 72 no 22 pp 1906ndash1913 2009

[2] K Franke and C Gaser ldquoLongitudinal changes in individualBrainAGE in healthy aging mild cognitive impairment andAlzheimerrsquos diseaserdquoGeroPsych vol 25 no 4 pp 235ndash245 2012

[3] MA IkramH A VroomanMWVernooij et al ldquoBrain tissuevolumes in the general elderly populationThe Rotterdam ScanStudyrdquo Neurobiology of Aging vol 29 no 6 pp 882ndash890 2008

[4] A Giorgio and N de Stefano ldquoClinical use of brain volumetryrdquoJournal of Magnetic Resonance Imaging vol 37 no 1 pp 1ndash142013

[5] M W Vannier R L Butterfield D Jordan W A Murphy RG Levitt and M Gado ldquoMultispectral analysis of magneticresonance imagesrdquo Radiology vol 154 no 1 pp 221ndash224 1985

[6] M A Balafar A R Ramli M I Saripan and S MashohorldquoReview of brain MRI image segmentation methodsrdquo ArtificialIntelligence Review vol 33 no 3 pp 261ndash274 2010

[7] J C Bezdek L O Hall and L P Clarke ldquoReview of MR imagesegmentation techniques using pattern recognitionrdquo MedicalPhysics vol 20 no 4 pp 1033ndash1048 1993

[8] L P Clarke R P Velthuizen M A Camacho et al ldquoMRIsegmentation methods and applicationsrdquo Magnetic ResonanceImaging vol 13 no 3 pp 343ndash368 1995

[9] J S Duncan X Papademetris J Yang M Jackowski X Zengand L H Staib ldquoGeometric strategies for neuroanatomic ana-lysis from MRIrdquo NeuroImage vol 23 supplement 1 pp S34ndashS45 2004

[10] D L PhamC Xu and J L Prince ldquoCurrentmethods inmedicalimage segmentationrdquoAnnual Review of Biomedical Engineeringvol 2 no 2000 pp 315ndash337 2000

[11] N Saeed ldquoMagnetic resonance image segmentation usingpattern recognition and applied to image registration andquantitationrdquo NMR in Biomedicine vol 11 no 4-5 pp 157ndash1671998

[12] J S Suri S Singh and L Reden ldquoComputer vision andpattern recognition techniques for 2-D and 3-D MR cerebralcortical segmentation (Part I) a state-of-the-art reviewrdquo PatternAnalysis amp Applications vol 5 no 1 pp 46ndash76 2002

[13] A P Zijdenbos and B M Dawant ldquoBrain segmentation andwhite matter lesion detection in MR imagesrdquo Critical Reviewsin Biomedical Engineering vol 22 no 5-6 pp 401ndash465 1994

[14] K Price ldquoAnything you can do I can do better (No you canrsquot)rdquoComputer Vision Graphics and Image Processing vol 36 no 2-3pp 387ndash391 1986

[15] R de Boer H A Vrooman M A Ikram et al ldquoAccuracy andreproducibility study of automatic MRI brain tissue segmenta-tion methodsrdquo NeuroImage vol 51 no 3 pp 1047ndash1056 2010

[16] J de Bresser M P Portegies A Leemans G J Biessels L JKappelle andM A Viergever ldquoA comparison ofMR based seg-mentation methods for measuring brain atrophy progressionrdquoNeuroImage vol 54 no 2 pp 760ndash768 2011

Computational Intelligence and Neuroscience 15

[17] M B Cuadra L Cammoun T Butz O Cuisenaire and J-P Thiran ldquoComparison and validation of tissue modelizationand statistical classification methods in T1-weighted MR brainimagesrdquo IEEE Transactions on Medical Imaging vol 24 no 12pp 1548ndash1565 2005

[18] F Klauschen A Goldman V Barra A Meyer-Lindenbergand A Lundervold ldquoEvaluation of automated brain MR imagesegmentation and volumetrymethodsrdquoHuman BrainMappingvol 30 no 4 pp 1310ndash1327 2009

[19] H Zaidi T Ruest F Schoenahl and M-L Montandon ldquoCom-parative assessment of statistical brain MR image segmentationalgorithms and their impact on partial volume correction inPETrdquo NeuroImage vol 32 no 4 pp 1591ndash1607 2006

[20] J A Frazier S M Hodge J L Breeze et al ldquoDiagnostic and sexeffects on limbic volumes in early-onset bipolar disorder andschizophreniardquo Schizophrenia Bulletin vol 34 no 1 pp 37ndash462008

[21] A Klein and J Tourville ldquo101 labeled brain images and aconsistent human cortical labeling protocolrdquo Frontiers in Neu-roscience vol 6 article 171 2012

[22] B Van Ginneken T Heimann and M Styner ldquo3D segmen-tation in the clinic a grand challengerdquo in Proceedings of the10th International Conference on Medical Image Computingand Computer Assisted Intervention (MICCAI rsquo07) pp 7ndash15Brisbane Australia October-November 2007

[23] A M Dale B Fischl and M I Sereno ldquoCortical surface-based analysis I Segmentation and surface reconstructionrdquoNeuroImage vol 9 no 2 pp 179ndash194 1999

[24] B Fischl M I Sereno and A M Dale ldquoCortical surface-basedanalysis II inflation flattening and a surface-based coordinatesystemrdquo NeuroImage vol 9 no 2 pp 195ndash207 1999

[25] M Jenkinson C F Beckmann T E J BehrensMWWoolrichand S M Smith ldquoFSLrdquoNeuroImage vol 62 no 2 pp 782ndash7902012

[26] J Ashburner and K J Friston ldquoUnified segmentationrdquo Neu-roImage vol 26 no 3 pp 839ndash851 2005

[27] Y D Reijmer A Leemans M Brundel L J Kappelle and GJ Biessels ldquoDisruption of the cerebral white matter network isrelated to slowing of information processing speed in patientswith type 2 diabetesrdquoDiabetes vol 62 no 6 pp 2112ndash2115 2013

[28] S Klein M Staring K Murphy M A Viergever and J P WPluim ldquoElastix a toolbox for intensity-based medical imageregistrationrdquo IEEE Transactions onMedical Imaging vol 29 no1 pp 196ndash205 2010

[29] K J Friston J T Ashburner S J Kiebel T E Nichols and WD Penny Eds Statistical Parametric Mapping The Analysis ofFunctional Brain Images Academic Press 2011

[30] F Ritter T Boskamp A Homeyer et al ldquoMedical imageanalysis a visual approachrdquo IEEE Pulse vol 2 no 6 pp 60ndash702011

[31] L R Dice ldquoMeasures of the amount of ecologic associationbetween speciesrdquo Ecology vol 26 no 3 pp 297ndash302 1945

[32] D P Huttenlocher G A Klanderman and W J RucklidgeldquoComparing images using the Hausdorff distancerdquo IEEE Trans-actions on Pattern Analysis and Machine Intelligence vol 15 no9 pp 850ndash863 1993

[33] A Alansary A Soliman F Khalifa et al ldquoMAP-based frame-work for segmentation of MR brain images based on visualappearance and prior shaperdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal September 2013

[34] A Jog S Roy J L Prince and A Carass ldquoMR brain segmenta-tion using decision treesrdquo in Proceedings of the MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal Nagoya Japan September 2013

[35] R Katyal S Paneri and M Kuse ldquoGaussian intensity modelwith neighborhood cues for fluid-tissue categorization ofmulti-sequence MR brain imagesrdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal Nagoya Japan September 2013

[36] QMahmoodMAlipoor A Chodorowski AMehnert andMPersson ldquoMultimodal MR brain segmentation using Bayesian-based adaptive mean-shift (BAMS)rdquo in Proceedings of theMICCAI WorkshopsmdashThe MICCAI Grand Challenge on MRBrain Image Segmentation (MRBrainS rsquo13) The Midas JournalSeptember 2013

[37] A van Opbroek F van der Lijn andM de Bruijne ldquoAutomatedbrain-tissue segmentation by multi-feature SVM classificationrdquoin Proceedings of the MICCAI WorkshopsmdashThe MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal September 2013

[38] S Pereira J Festa J A Mariz N Sousa and C A SilvaldquoAutomatic brain tissue segmentation of multi-sequence MRimages using random decision forestsrdquo in Proceedings of theMICCAI Grand Challenge on MR Brain Image Segmentation(MRBrainS rsquo13) The Midas Journal Nagoya Japan September2013

[39] MRajchl J Baxter J Yuan T Peters andAKhan ldquoMulti-atlas-based segmentation with hierarchical max-flowrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[40] D Sarikaya L Zhao and J J Corso ldquoMulti-atlas brain MRIsegmentation with multiway cutrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[41] H Vrooman F van der Lijn and W Niessen ldquoAuto-kNNbrain tissue segmentation using automatically trained k-nearest-neighbor classificationrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[42] S Vyas P Burlina D Kleissas and R Mukherjee ldquoAutomatedwalks using machine learning for segmentationrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[43] C Wang and O Smedby ldquoFully automatic brain segmentationusing model-guided level sets and skeleton-based modelsrdquo inProceedings of theMICCAI Grand Challenge onMRBrain ImageSegmentation (MRBrainS rsquo13) The Midas Journal NagoyaJapan September 2013

[44] C Cortes and V Vapnik ldquoSupport-vector networksrdquo MachineLearning vol 20 no 3 pp 273ndash297 1995

[45] H A Vrooman C A Cocosco F van der Lijn et al ldquoMulti-spectral brain tissue segmentation using automatically trainedk-Nearest-Neighbor classificationrdquo NeuroImage vol 37 no 1pp 71ndash81 2007

[46] C Wang H Frimmel and O Smedby ldquoFast level-set basedimage segmentation using coherent propagationrdquo MedicalPhysics vol 41 no 7 Article ID 073501 2014

16 Computational Intelligence and Neuroscience

[47] A Jog S Roy A Carass and J L Prince ldquoMagnetic resonanceimage synthesis through patch regressionrdquo in Proceedings of theIEEE 10th International Symposium onBiomedical Imaging (ISBIrsquo13) pp 350ndash353 April 2013

[48] J G Sied A P Zijdenbos and A C Evans ldquoA non-parametricmethod for automatic correction of intensity non-uniformity inMRI datardquo IEEE Transactions on Medical Imaging vol 17 no 1pp 87ndash97 1998

[49] A Carass J Cuzzocreo M B Wheeler P-L Bazin S MResnick and J L Prince ldquoSimple paradigm for extra-cerebraltissue removal algorithm and analysisrdquoNeuroImage vol 56 no4 pp 1982ndash1992 2011

[50] M Modat G R Ridgway Z A Taylor et al ldquoFast free-form deformation using graphics processing unitsrdquo ComputerMethods and Programs in Biomedicine vol 98 no 3 pp 278ndash284 2010

[51] M Rajchl J Yuan J A White et al ldquoInteractive hierarchical-flow segmentation of scar tissue from late-enhancement cardiacmr imagesrdquo IEEE Transactions on Medical Imaging vol 33 no1 pp 159ndash172 2014

[52] QMahmood A Chodorowski AMehnert andM Persson ldquoAnovel Bayesian approach to adaptive mean shift segmentationof brain imagesrdquo in Proceedings of the 25th IEEE InternationalSymposiumonComputer-BasedMedical Systems (CBMS rsquo12) pp1ndash6 June 2012

[53] A El-Baz M Nitzken A Alansary A Soliman and MCasanova ldquoBrain Segmentation Method for Young Childrenand Adultsrdquo US Provisional Patent Application 13100

[54] A Alansary A Soliman M Nitzken et al ldquoAn integratedgeometrical and stochastic approach for accurate infant brainextractionrdquo in Proceedings of the IEEE International Conferenceon Image Processing (ICIP rsquo14) pp 3542ndash3546 October 2014

[55] V Popescu M Battaglini W S Hoogstrate et al ldquoOptimizingparameter choice for FSL-Brain Extraction Tool (BET) on 3DT1 images in multiple sclerosisrdquo NeuroImage vol 61 no 4 pp1484ndash1494 2012

[56] J L Whitwell W R Crum H C Watt and N C FoxldquoNormalization of cerebral volumes by use of intracranial vol-ume implications for longitudinal quantitative MR imagingrdquoAmerican Journal of Neuroradiology vol 22 no 8 pp 1483ndash1489 2001

[57] S M Smith ldquoFast robust automated brain extractionrdquo HumanBrain Mapping vol 17 no 3 pp 143ndash155 2002

[58] D W Shattuck G Prasad M Mirza K L Narr and A WToga ldquoOnline resource for validation of brain segmentationmethodsrdquo NeuroImage vol 45 no 2 pp 431ndash439 2009

[59] I Isgum M J N L Benders B Avants et al ldquoEvaluation ofautomatic neonatal brain segmentation algorithms the Neo-BrainS12 challengerdquo Medical Image Analysis vol 20 no 1 pp135ndash151 2015

[60] P Moeskops M J N L Benders S M Chita et al ldquoAutomaticsegmentation of MR brain images of preterm infants usingsupervised classificationrdquo NeuroImage 2015

[61] P Moeskops M A Viergever M J N L Benders and IIsgum ldquoEvaluation of an automatic brain segmentationmethoddeveloped for neonates on adult MR brain imagesrdquo in MedicalImaging Image Processing vol 9413 of Proceedings of SPIEInternational Society for Optics and Photonics Orlando FlaUSA February 2015

[62] L Wang Y Gao F Shi et al ldquoLINKS learning-based multi-source IntegratioN frameworK for Segmentation of infant brainimagesrdquo NeuroImage vol 108 pp 160ndash172 2015

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

2 Computational Intelligence and Neuroscience

segment GM WM and CSF and evaluated using three evaluation metrics (Dice H95 and AVD) and the results are published onthe MRBrainS13 website We present the results of eleven segmentation algorithms that participated in the MRBrainS13 challengeworkshop at MICCAI where the framework was launched and three commonly used freeware packages FreeSurfer FSL andSPMTheMRBrainS evaluation framework provides an objective and direct comparison of all evaluated algorithms and can aid inselecting the best performing method for the segmentation goal at hand

1 Introduction

Multiple large population studies [1ndash3] have shown theimportance of quantifying brain structure volume forexample to detect or predict small vessel disease andAlzheimerrsquos disease In clinical practice brain volumetry canbe of value in disease diagnosis progression and treatmentmonitoring of a wide range of neurologic conditionssuch as Alzheimerrsquos disease dementia focal epilepsyParkinsonism and multiple sclerosis [4] Automatic brainstructure segmentation in MRI dates back to 1985 [5] andmany methods have been proposed since then Howeverthe multitude of methods proposed [6ndash13] complicatesthe choice for a certain method above others As earlyas 1986 Price [14] stressed the importance of comparingdifferent approaches to the same type of problem Variousstudies have addressed this issue and evaluated differentbrain structure segmentation methods [15ndash19] Howeverseveral factors complicate direct comparison of differentapproaches Not all algorithms are publicly available andif they are researchers who use them are generally not asexperienced with these algorithms as they are with their ownalgorithm in terms of parameter tuning which could resultin a bias towards their own method This problem does notexist when researchers apply their own method to publiclyavailable data Therefore publicly available databases likethe ldquoAlzheimerrsquos Disease Neuroimaging Initiativerdquo (ADNI)(httpadniloniuscedu) the ldquoInternet Brain SegmentationRepositoryrdquo (IBSR) (httpwwwnitrcorgprojectsibsr) theCANDI Share Schizophrenia Bulletin 2008 (httpswwwnitrcorgprojectscs schizbull08) [20] and Mindboggle(httpwwwmindboggleinfo) [21] are important initiativesto enable comparison of various methods on the same dataHowever due to the use of subsets of the available data anddifferent evaluation measures direct comparison can be pro-blematic To address this issue grand challenges in bio-medical image analysis were introduced in 2007 [22]Participants in these competitions can apply their algorithmsto the provided data after which their results are evaluatedand ranked by the organizersMany challenges (httpgrand-challengeorgAll Challenges) have been organized sincethen providing an insight into the performance of automaticalgorithms for specific tasks in medical image analysis

In this paper we introduce the MRBrainS challenge eval-uation framework (httpmrbrains13isiuunl) an onlineframework to evaluate automatic and semiautomatic algo-rithms that segment gray matter (GM) white matter (WM)and cerebrospinal fluid (CSF) in 3T brain MRI scans ofolder (mean age 71) subjects with varying degrees of atrophyand white matter lesions This framework has three mainadvantages Firstly researchers apply their own segmentationalgorithms to the provided data Parameters are optimally

tuned to achieve the best possible performance Secondlyall algorithms are applied to the exact same data and thereference standard of the test data is unknown to the par-ticipating researchers Thirdly the evaluation algorithm andmeasures are the same for all evaluated algorithms enablingdirect comparison of the various algorithms The frameworkwas launched at the MRBrainS13 challenge workshop at theMedical Image Computing and Computer Assisted Interven-tion (MICCAI) conference on September 26th in 2013 Eleventeams participated in the challenge workshop with a widevariety of segmentation algorithms the results for which arepresented in this paper and provide a benchmark for theproposed evaluation framework In addition we evaluatedthree commonly used freeware packages on the evaluationframework FreeSurfer (httpsurfernmrmghharvardedu)[23 24] FSL (httpfslfmriboxacukfslfslwiki) [25] andSPM (httpwwwfilionuclacukspm) [26]

2 Materials and Methods

21 Evaluation Framework TheMRBrainS evaluation frame-work is set up as follows Multisequence (T1-weighted T1-weighted inversion recovery and T2-weighted fluid atten-uated inversion recovery) 3T MRI scans of twenty sub-jects are available for download on the MRBrainS website(httpmrbrains13isiuunl) The data is described in moredetail in Section 211 All scans were manually segmentedinto GM WM and CSF These manual segmentations areused as the reference standard for the evaluation frameworkThe annotation process for obtaining the reference standardis described in Section 212 For five of the twenty datasetsthe reference standard is provided on the website and can beused for training an automatic segmentation algorithm Theremaining fifteen MRI datasets have to be segmented by theparticipating algorithms into GM WM and CSF For thesefifteen datasets the reference standard is not provided onlineThe segmentation results can be submitted on the MRBrainSwebsite With each submission a short description of thesegmentation algorithm has to be provided which shouldat least describe the algorithm the used MRI sequenceswhether the algorithm is semi- or fully automatic and theaverage runtime of the algorithm The segmentation resultsare then evaluated (Section 213) and ranked (Section 214)by the organizers and the results are presented on the websiteMore information on how to use the evaluation frameworkis provided in the details section of the MRBrainS website(httpmrbrains13isiuunldetailsphp)

211 Data The focus was on brain segmentation in thecontext of ageing Twenty subjects (mean age plusmn SD = 71 plusmn 4years 10 male 10 female) were selected from an ongoing

Computational Intelligence and Neuroscience 3

cohort study of older (65ndash80 years of age) functionallyindependent individuals without a history of invalidatingstroke or other brain diseases [27] This study was approvedby the local ethics committee of the University MedicalCenter Utrecht (Netherlands) and all participants signed aninformed consent form To be able to test the robustnessof the segmentation algorithms in the context of ageing-related pathology the subjects were selected to have varyingdegrees of atrophy and white matter lesions Scans withmajor artefacts were excluded MRI scans were acquired ona 30 T Philips AchievaMR scanner at the University MedicalCenter Utrecht (Netherlands) The following sequences wereacquired and used for the evaluation framework 3D T1 (TR79ms TE 45ms) T1-IR (TR 4416ms TE 15ms and TI400ms) and T2- FLAIR (TR 11000ms TE 125ms and TI2800ms) Since the focus of the MRBrainS evaluation frame-work is on comparing different segmentation algorithms weperformed two preprocessing steps to limit the influenceof different registration and bias correction algorithms onthe segmentation results The sequences were aligned byrigid registration using Elastix [28] and bias correction wasperformed using SPM8 [29] After registration the voxel sizewithin all provided sequences (T1 T1 IR and T2 FLAIR)was 096 times 096 times 300mm3 The original 3D T1 sequence(voxel size 10 times 10 times 10mm3) was provided as well Fivedatasets that were representative for the overall data (2 male3 female varying degrees of atrophy andwhitematter lesions)were selected for training The remaining fifteen datasets areprovided as test data

212 Reference Standard Manual segmentations were per-formed to obtain a reference standard for the evaluationframework All axial slices of the 20 datasets (096 times 096times 300mm3) were manually segmented by trained researchassistants in a darkened room with optimal viewing con-ditions All segmentations were checked and corrected bythree experts a neurologist in training a neuroradiologistin training and a medical image processing scientist Toperform the manual segmentations an in-house developedtool based on MeVisLab (MeVis Medical Solutions AGBremen Germany) was used employing a freehand splinedrawing technique [30] The closed freehand spline drawingtechnique was used to delineate the outline of each brainstructure starting at the innermost structures (Figure 1(a))working outward The closed contours were converted tohard segmentations and the inner structures were itera-tively subtracted from the outer structures to construct thefinal hard segmentation image (Figure 1(b)) The followingstructures were segmented and are available for trainingcortical gray matter (1) basal ganglia (2) white matter (3)white matter lesions (4) peripheral cerebrospinal fluid (5)lateral ventricles (6) cerebellum (7) and brainstem (8)Thesestructures can be merged into gray matter (1 2) white matter(3 4) and cerebrospinal fluid (5 6) The cerebellum andbrainstem are excluded from the evaluation All structureswere segmented on theT1-weighted scans thatwere registeredto the FLAIR scans except for the white matter lesions(WMLs) and the CSF outer border (used to determine

the intracranial volume) The WMLs were segmented onthe FLAIR scan by the neurologist in training and checkedand corrected by the neuroradiologist in training The CSFouter border was segmented using both the T1-weightedand the T1-weighted IR scan since the T1-weighted IR scanshows higher contrast at the borders of the intracranialvolumeTheCSF segmentation includes all vessels (includingthe superior sagittal sinus and the transverse sinuses) andnonbrain structures such as the cerebral falx and choroidplexuses

213 Evaluation To evaluate the segmentation results weuse three types of measures a spatial overlap measure aboundary distance measure and a volumetric measure TheDice [31] coefficient is used to determine the spatial overlapand is defined as

119863 =2 |119860 cap 119866|

|119860| + |119866|

sdot 100 (1)

where 119860 is the segmentation result 119866 is the referencestandard and 119863 is the Dice expressed as percentages The95th-percentile of theHausdorffdistance is used to determinethe distance between the segmentation boundaries Theconventional Hausdorff distance uses the maximum whichis very sensitive to outliers To correct for outliers we usethe 95th-percentile of theHausdorff distance by selecting the119870th ranked distance as proposed by Huttenlocher et al [32]

ℎ95 (119860 119866) =

95119870

th119886isin119860

min119892isin119866

1003817100381710038171003817119892 minus 1198861003817100381710038171003817 (2)

where 95119870th119886isin119860

is the119870th rankedminimumEuclidean distancewith 119870119873

119886= 95 119860 is the set of boundary points

1198861 119886

119873119886

of the segmentation result and 119866 is the set ofboundary points 119892

1 119892

119873119892

of the reference standard The95th-percentile of the Hausdorff distance is defined as

11986795(119860 119866) = max (ℎ

95(119860 119866) ℎ

95(119866 119860)) (3)

The third measure is the percentage absolute volume differ-ence defined as

AVD =10038161003816100381610038161003816119881119886minus 119881119892

10038161003816100381610038161003816

119881119892

sdot 100 (4)

where 119881119886is the volume of the segmentation result and 119881

119892

is the volume of the reference standard These measures areused to evaluate the following brain structures in each of thefifteen test datasets GM WM CSF brain (GM + WM) andintracranial volume (GM + WM + CSF) The brainstem andcerebellum are excluded from the evaluation

214 Ranking To compare the segmentation algorithmsthat participate in the MRBrainS evaluation framework thealgorithms are ranked based on their overall performance tosegment GMWM and CSF Each of these components (119862 =GMWMCSF) is evaluated by using the three evaluationmeasures (119872 = 119863119867

95AVD) described in Section 213

4 Computational Intelligence and Neuroscience

(a) (b) (c) (d) (e)

Figure 1 Example of themanually drawn contours (a) the resulting hard segmentationmap (GM light blueWM yellow andCSF dark blue)that is used as the reference standard (b) the T1-weighted scan (c) the T1-weighted inversion recovery (IR) scan (d) and the T2-weightedfluid attenuated inversion recovery (FLAIR) scan (e)

For each component 119888 isin 119862 and each evaluation measure119898 isin119872 the mean and standard deviation are determined over all15 test datasets The segmentation algorithms are then sortedon the mean 119863 value in descending order and on the mean11986795

and AVD value in ascending order Each segmentationalgorithm receives a rank (119903) between 1 (ranked best) and 119899(number of participating algorithms) for each component 119888and each evaluation measure 119898 The final ranking is basedon the overall score of each algorithm which is the sum overall ranks defined as

119904 =

|119872|

sum

119898=0

|119862|

sum

119888=0

119903119898119888 (5)

where 119903119898119888

is the rank of the segmentation algorithm formeasure 119898 of component 119888 For the final ranking 119903 theoverall scores 119904 are sorted in ascending order and rankedfrom 1 to 119899 In case two or more algorithms have equalscores the standard deviation over all 15 test datasets is takeninto account to determine the final rank The segmentationalgorithms are then sorted on the standard deviation inascending order and ranked for each component 119888 and eachevaluation measure119898 The overall score is determined using(5) and the algorithms are sorted based on this score inascending order and ranked from 1 to 119899 The algorithms thathave equal overall scores based on the mean value are thenranked based on this standard deviation rank

22 EvaluatedMethods Theevaluation framework describedin Section 21 was launched at the MRBrainS13 challengeworkshop at the Medical Image Computing and ComputerAssisted Intervention (MICCAI) conference on September26th in 2013 For the workshop challenge the test datasetswere split into twelve off-site and three on-site test datasetsFor the off-site part teams could register on the MRBrainSwebsite (httpmrbrains13isiuunl) and download the fivetraining and twelve test datasets A time slot of eight weekswas available for teams to download the data train theiralgorithms segment the test datasets and submit their resultson the website Fifty-eight teams downloaded the data ofwhich twelve submitted their segmentation results The eval-uation results were reported to the twelve teams and all teamssubmitted a workshop paper to the MRBrainS13 challengeworkshop at MICCAI Eleven teams presented their results

at the workshop and segmented the three on-site test datasetslive at the workshop within a time slot of 35 hours Thesealgorithms provide a benchmark for the proposed evaluationframework and are briefly described in Sections 221ndash2211in alphabetical order of teamsrsquo names The teamsrsquo namesare used in the paper to identify the methods For a fulldescription of the methods we refer to the workshop papers[33ndash43] In Section 2212 we describe the evaluated freewarepackages

221 BIGR2 [37] This multifeature SVM [37] method clas-sifies voxels by using a Support Vector Machine (SVM)classifier [44] with a Gaussian kernel Besides spatial featuresand intensity information from all three MRI sequences theSVM classifier incorporates Gaussian-scale-space features tofacilitate a smooth segmentation Skull stripping is performedby nonrigid registration of the masks of the training imagesto the target image

222 Bigr neuro [41] This auto-kNN [45] method is basedon an automatically trained kNN-classifier First a proba-bilistic tissue atlas is generated by nonrigidly registering themanually annotated atlases to the subject of interest Trainingsamples are obtained by thresholding the probabilistic atlasand subsequently pruning the feature space White matterlesions are detected by applying an adaptive threshold deter-mined from the tissue segmentation to the FLAIR sequence

223 CMIV [43] A statistical-model-guided level-setmethod is used to segment the skull brain ventricles andbasal ganglia Then a skeleton-based model is created byextracting the midsurface of the gray matter and definingthe thickness This model is incorporated into a level-setframework to guide the cortical gray matter segmentationThe coherent propagation algorithm [46] is used to acceleratethe level-set evolution

224 Jedi MindMeld [42] Thismethod starts by preprocess-ing the data via anisotropic diffusion For each 2D slice of alabeled dataset the canny edge pixels are extracted and theTourist Walk is computed This is done for axial sagittal andcoronal views Machine learning is used with these featuresto automatically label edge pixels in an unlabeled dataset

Computational Intelligence and Neuroscience 5

Finally these labels are used by the Random Walker forautomatic segmentation

225 LNMBrains [35] The voxel intensities of all MRIsequences are modelled as a Gaussian distribution for eachlabel The parameters of the Gaussian distributions areevaluated asmaximum likelihood estimates and the posteriorprobability of each label is determined by using Bayesianestimation A feature set consisting of regional intensitytexture spatial location of voxels and the posterior prob-ability estimates is used to classify each voxel into CSFWM GM or background by using a multicategory SVMclassifier

226 MNAB [38] This method uses Random DecisionForests to classify the voxels into GM WM and CSF Itstarts by a skull stripping procedure followed by an intensitynormalization of each MRI sequence Feature extraction isthen performed on the intensities posterior probabilitiesneighborhood statistics tissue atlases and gradient mag-nitude After classification isolated voxels are removed bypostprocessing

227 Narsil [34] This is a model-free algorithm that usesensembles of decision trees [47] to learn the mapping fromimage features to the corresponding tissue label The ensem-bles of decision trees are constructed from correspondingimage patches of the provided T1 and FLAIR scans withmanual segmentations The N3 algorithm [48] was used foradditional inhomogeneity correction and SPECTRE [49] wasused for skull stripping

228 Robarts [39] Multiatlas registration [50] with the T1training images was used to propagate labels to generatesample histograms in a log-likelihood intensity model andprobabilistic shape priors These were employed in a MAPdata term and regularized via computation of a hierar-chical max-flow [51] A brain mask from registration ofthe T1-IR training images was used to obtain the finalresults

229 S2 QM [36] This method [52] is based on Bayesian-based adaptive mean shift and the voxel-weighted 119870-meansalgorithm The former is used to segment the brain into alarge number of clusters or modes The latter is employed toassign these clusters to one of the three components WMGM or CSF

2210 UB VPMLMed [40] This method creates a multiatlasby registering the training images to the subject imageand then propagating the corresponding labels to a fullyconnected graph on the subject image Label fusion thencombines the multiple labels into one label at each voxelwith intensity similarity based weighted voting Finally themethod clusters the graph using multiway cut in order toachieve the final segmentation

2211 UofL BioImaging [33] This is an automated MAP-based method aimed at unsupervised segmentation of differ-ent brain tissues from T1-weighted MRI It is based on theintegration of a probabilistic shape prior a first-order inten-sity model using a Linear Combination of Discrete Gaussians(LCDG) and a second-order appearance model These threefeatures are integrated into a two-level joint Markov-GibbsRandom Field (MGRF) model of T1-MR brain imagesSkull stripping was performed using BET2 [40] followed byan adaptive threshold-based technique to restore the outerborder of the CSF using both T1 andT1-IR this techniquewasnot described in [33] due to a US patent application [53] butis described in [54] This method was applied semiautomat-ically to the MRBrainS test data due to per scan parametertuning

2212 Freeware Packages Next to the methods evaluatedat the workshop we evaluated three commonly used free-ware packages for MR brain image segmentation FreeSurfer(httpsurfernmrmghharvardedu) [23 24] FSL (httpfslfmriboxacukfslfslwiki) [25] and SPM12 (httpwwwfilionuclacukspm) [26] All packages were applied usingthe default settings unless mentioned otherwise FreeSurfer(v530) was applied to the high resolution T1 sequence Themri label2vol tool was used to map the labels on the thickslice T1 that was used for the evaluation FSL (v50) wasdirectly applied to the thick slice T1 and provides both apveseg and a seg file as binary output We evaluated bothof these files The fractional intensity threshold parameterldquo119891rdquo of the BET tool that sets the brainnonbrain intensitythreshold was set according to [55] at 02 (Philips Achieva3T setting) SPM12 was directly applied to the thick sliceT1 sequence as well However it also provides the optionto add multiple MRI sequences Therefore we evaluatedSPM12 not only on the thick slice T1 sequence but addedthe T1-IR and the T2-FLAIR scan as well and tested variouscombinationsThe number of Gaussians was set according tothe SPM manual to two for GM two for WM and two forCSF

23 Statistical Analysis All evaluated methods werecompared to the reference standard In summary ofthe results the mean and standard deviation overall 15 test datasets were calculated per component(GM WM and CSF) and combination of components(brain intracranial volume) and per evaluation measure(Dice 95th-percentile Hausdorff distance and absolutevolume difference) for each of the evaluated methodsBoxplots were created using R version 303 (R projectfor statistical computing (httpwwwr-projectorg))Since white matter lesions should be segmented as whitematter the percentage of white matter lesion voxelssegmented as white matter (sensitivity) was calculatedfor each algorithm over all 15 test datasets to evaluatethe robustness of the segmentation algorithms againstpathology

6 Computational Intelligence and Neuroscience

3 Results

Table 1 presents the final ranking (119903) of the evaluatedmethodsthat participated in the workshop as well as the evalu-ated freeware packages During the workshop team UofLBioImaging ranked first and BIGR2 ranked second with onepoint difference in the overall score 119904 (5) However addingthe results of the freeware packages resulted in an equal scorefor UofL BioImaging and BIGR2 Therefore the standarddeviation rank was taken into account and BIGR2 is rankedfirst with standard deviation rank four and UofL BioImagingis ranked second with standard deviation rank eight Table 1further presents the mean standard deviation and rank foreach evaluation measure (119863119867

95 and AVD) and component

(GM WM and CSF) as well as the brain (WM + GM) andintracranial volume (WM+GM+CSF) Team BIGR2 scoredbest for theGMWM and brain segmentation and teamUofLBioImaging for the CSF segmentation Team Robarts scoredbest for the intracranial volume segmentation The boxplotsfor all evaluation measures and components are shown inFigures 2ndash4 and include the results of the freeware packagesFigure 5 shows an example of the segmentation results atthe height of the basal ganglia (slice 22 of test subject 9)The sensitivity of the algorithms to segment white matterlesions as WM and examples of the segmentation results inthe presence of white matter lesions (slice 31 of test subject3) are shown in Figure 6 Team UB VPML Med scores thehighest sensitivity of white matter lesions segmented as whitematter and is therefore most robust in the presence of thistype of pathology

4 Discussion

In this paper we proposed the MRBrainS challenge onlineevaluation framework to evaluate automatic and semiauto-matic algorithms for segmenting GM WM and CSF on3T multisequence (T1 T1-IR and T2-FLAIR) MRI scans ofthe brain We have evaluated and presented the results ofeleven segmentation algorithms that provide a benchmarkfor algorithms that will use the online evaluation frameworkto evaluate their performance Team UofL BioImaging andBIGR2 have equal overall scores but BIGR2 was rankedfirst based on the standard deviation ranking The evalu-ated methods represent a wide variety of algorithms thatinclude Markov random field models clustering approachesdeformable models and atlas-based approaches and classi-fiers (SVM KNN and decision trees) The presented evalua-tion framework provides an insight into the performance ofthese algorithms in terms of accuracy and robustness Variousfactors influence the choice for a certain method aboveothers We provide three measures that could aid in selectingthe method that is most appropriate for the segmentationgoal at hand a boundarymeasure (95th-percentile Hausdorffdistance 119867

95) an overlap measure (Dice coefficient 119863) and

a volume measure (absolute volume difference AVD) Allthree measures are taken into account for the final rankingof the methods This ranking was designed to get a quickinsight into how the methods perform in comparison to eachother The best overall method is the method that performs

well for all three measures and all three components (GMWM andCSF) However whichmethod to select depends onthe segmentation goal at hand Not all measures are relevantfor all segmentation goals For example if segmentation isused for brain volumetry [4] the overlap (119863) and volume(AVD) measures of the brain and intracranial volume (usedfor normalization [56]) segmentations are important to takeinto account On the other hand if segmentation is usedfor cortical thickness measurements the focus should be onthe gray matter boundary (119867

95) and overlap (119863) measures

Therefore the final ranking should be used to get a first insightinto the overall performance after which the performanceof the measures and components that are most relevant forthe segmentation goal at hand should be considered Besidesaccuracy robustness could also influence the choice for acertain method above others For example team UB VPMLMed shows a high sensitivity score for segmenting whitemat-ter lesions as white matter (Figure 6) and shows a consistentsegmentation performance of gray and white matter over all15 test datasets (Figures 2ndash4) This could be beneficial forsegmenting scans of populations with white matter lesionsbut is less important if the goal is to segment scans ofyoung healthy subjects In the latter case the most accuratesegmentation for gray and white matter (team BIGR2) ismore interesting If a segmentation algorithm is to be usedin clinical practice speed is an important consideration aswell The runtime of the evaluated methods is reported inTable 1 However these runtimes are merely an indicationof the required time since academic software is generallynot optimized for speed and the runtime is measured ondifferent computers andplatformsAnother relevant aspect ofthe evaluation framework is the comparison of multi- versussingle-sequence approaches For example most methodsstruggle with the segmentation of the intracranial volume onthe T1-weighted scan There is no contrast between the CSFand the skull and the contrast between the dura mater andthe CSF is not always sufficient Team Robarts used an atlas-based registration approach on the T1-IR scan (good contrastbetween skull and CSF) to segment the intracranial volumewhich resulted in the best performance for intracranialvolume segmentation (Table 1 Figures 2ndash4) Most methodsadd the T2-FLAIR scan to improve robustness against whitematter lesions (Table 1 Figure 6) Although using only theT1-weighted scan and incorporating prior shape information(team UofL BioImaging) can be very effective also thefreeware packages support this as well Since FreeSurfer isan atlas-based method it uses prior information and is themost robust of all freeware packages to white matter lesionsHowever adding the T2 FLAIR scan to SPM12 increasesrobustness against white matter lesions as well as comparedto applying SPM12 to the T1 scan only (Figure 6) In generalSPM12with the T1 and the T2-FLAIR sequence performswellin comparison to the other freeware packages (Table 1 andFigures 2ndash4) on the thick slice MRI scans Although addingthe T1-IR scan to SPM increases the performance of the CSFand ICV segmentations as compared to using only the T1 andT2-FLAIR sequence it decreases the performance of the GMand WM segmentations Therefore adding all sequences toSPM12 did not result in a better overall performance

Computational Intelligence and Neuroscience 7

Table 1 Results of the 11 evaluated algorithms presented at the workshop and the evaluated freeware packages on the 15 test datasets Thealgorithms are ranked (119903) based on their overall score (119904) by using (5) This score is based on the ranks of the gray matter (GM) white matter(WM) and cerebrospinal fluid (CSF) segmentation and the three evaluated measures Dice coefficient (119863 in ) 95th-percentile Hausdorffdistance (119867

95in mm) and the absolute volume difference (AVD in ) The rank 119903

119898119888denotes the rank based on the mean (120583) over all 15 test

datasets for each measure 119898 (0 119863 1 11986795 and 2 AVD) and component 119888 (0 GM 1 WM 2 CSF 3 brain (WM + GM) and 4 intracranial

volume (ICV = WM + GM + CSF)) Teams BIGR2 and UofL BioImaging and FreeSurfer and Jedi Mind Meld have equal scores based onthe mean (120583) therefore the ranking based on the standard deviation (120590) is taken into account to determine the final rank (BIGR2 120590 rank 4UofL BioImaging 120590 rank 8 FreeSurfer 120590 rank 13 and Jedi MindMeld 120590 rank 17) Columns 2 and 3 present the average runtime 119905 per scan inseconds (s) minutes (m) or hours (h) and the scans (T1 T1-weighted scan 3D T1 3D T1-weighted scan IR T1-weighted inversion recovery(IR) and F T1-weighted FLAIR) that are used for processing

119903 Team 119905 Scans

GM WM CSF

119904

Brain ICV119903001199031011990320119903011199031111990321119903021199031211990322

119903031199031311990323119903041199031411990324

119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590)

1 BIGR2 35m T1 IR F1 2 4 2 2 4 4 5 14

381 2 12 8 4 12

847 19 61 884 24 60 783 32 23 951 27 32 960 39 52(13) (04) (33) (12) (05) (51) (50) (08) (17) (05) (08) (16) (13) (11) (30)

2 UofLBioImaginglowast 6 s T1

5 1 9 4 1 13 2 2 138

2 1 12 5 2 5830 17 86 879 22 87 789 27 97 949 24 39 967 34 18(15) (03) (54) (20) (06) (66) (42) (05) (10) (06) (05) (20) (08) (06) (20)

3 CMIV 3m T1 F6 7 5 5 3 10 3 3 8

505 7 2 4 3 11

824 27 68 877 24 73 786 30 14 945 38 26 968 38 49(14) (04) (40) (16) (04) (38) (31) (04) (59) (05) (11) (22) (08) (13) (23)

4 UB VPMLMed 30m T1 IR F

4 4 2 1 5 7 8 13 1761

4 4 1 10 11 15833 21 59 886 27 71 748 43 31 946 28 24 948 66 77(13) (03) (53) (17) (04) (38) (71) (17) (19) (06) (04) (18) (20) (20) (41)

5 Bigr neuro 2 h T1 F7 13 3 6 6 9 6 4 10

647 10 10 7 5 8

815 37 59 873 30 73 782 32 16 940 46 36 963 39 35(17) (09) (42) (14) (04) (38) (47) (06) (14) (08) (14) (24) (12) (09) (27)

6 Robarts 16m 3D T1 IR11 3 15 8 8 6 1 1 13

6616 3 18 1 1 1

797 20 98 862 31 71 803 27 20 931 28 79 979 26 09(24) (01) (73) (13) (04) (62) (41) (05) (13) (16) (05) (36) (03) (04) (07)

7 Narsil 2m T1 F3 5 1 7 11 2 17 18 7

713 5 3 16 18 9

835 23 55 871 33 58 666 133 14 948 29 29 925 24 37(18) (04) (44) (13) (09) (53) (24) (54) (95) (05) (05) (20) (05) (89) (17)

8 SPM T1 F 3m T1 F8 9 16 9 7 1 9 14 2

759 14 14 6 14 4

812 29 10 860 30 52 741 46 10 939 58 53 966 82 15(22) (03) (85) (15) (01) (38) (34) (06) (47) (10) (22) (38) (02) (30) (10)

9 SPM T1 IR 3m T1 IR12 11 7 16 12 5 5 10 3

818 11 7 2 8 2

794 30 72 835 36 63 783 40 10 939 46 34 977 65 10(21) (04) (63) (21) (03) (46) (38) (06) (57) (08) (12) (28) (02) (13) (08)

10 MNAB 15m T1 IR F2 8 12 3 4 11 15 15 16

866 9 11 15 12 17

839 28 91 880 27 78 681 49 29 945 45 38 925 71 97(21) (09) (65) (12) (08) (40) (40) (22) (21) (10) (20) (32) (11) (42) (47)

11 SPM T1 3m T19 10 6 11 10 3 11 16 15

9110 8 6 9 13 13

803 30 69 856 31 60 707 53 23 939 44 32 953 81 55(24) (05) (68) (17) (01) (41) (38) (15) (157) (09) (16) (29) (09) (37) (37)

12 FSL Seg 10m T113 16 10 10 13 14 12 6 5

9913 13 4 12 6 6

787 43 86 860 37 115 699 34 12 933 55 30 942 53 34(22) (12) (63) (26) (08) (63) (28) (02) (103) (08) (14) (15) (08) (11) (15)

8 Computational Intelligence and Neuroscience

Table 1 Continued

119903 Team 119905 Scans

GM WM CSF

119904

Brain ICV119903001199031011990320119903011199031111990321119903021199031211990322

119903031199031311990323119903041199031411990324

119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590)

13 SPM T1 IR F 4m T1 IR F10 12 18 15 14 12 7 11 6

10512 15 13 3 15 3

801 30 139 836 38 84 769 41 12 936 59 51 977 82 12(24) (02) (96) (21) (05) (52) (31) (05) (60) (11) (18) (36) (02) (18) (09)

14 FSL PVSeg 10m T115 15 8 13 15 17 13 7 4

10711 16 9 13 7 7

777 43 84 848 38 197 695 34 11 936 61 35 942 53 34(26) (13) (65) (32) (09) (10) (22) (03) (58) (09) (16) (30) (08) (11) (15)

15 FreeSurfer 1 h 3D T116 6 17 12 9 8 18 12 18

11617 6 16 17 10 18

774 23 121 852 31 72 658 43 50 923 36 63 924 66 10(20) (06) (60) (22) (05) (46) (37) (06) (196) (08) (06) (34) (09) (04) (45)

16 Jedi MindMeld 27 s T1 IR F

14 14 11 17 16 15 10 8 11116

15 12 8 11 16 14778 39 89 806 45 137 733 37 18 932 54 35 943 20 68(59) (28) (76) (96) (36) (12) (35) (08) (90) (17) (48) (30) (17) (37) (31)

17 S2 QM 15 h T1 IR F17 17 13 14 17 18 16 9 9

13014 17 15 14 9 10

764 55 93 839 49 248 679 38 14 933 70 55 937 65 40(34) (30) (64) (34) (32) (10) (23) (06) (99) (14) (35) (45) (11) (14) (22)

18 LNMBrains 5m T1 IR F18 18 14 18 18 16 14 17 12

14518 18 17 18 17 16

728 68 95 783 68 153 688 77 20 885 86 74 892 204 79(53) (23) (76) (64) (35) (13) (66) (24) (13) (45) (28) (77) (48) (37) (83)

lowastSemiautomatic due to per scan parameter tuning

Besides the advantages of the MRBrainS evaluationframework there are some limitations that should be takeninto account The T1-weighted IR and the T2-weightedFLAIR scan were acquired with a lower resolution (096times 096 times 300mm3) than the 3D T1-weighted scan (10 times10 times 10mm3) To be able to provide a registered multise-quence dataset the 3D T1-weighted scan was registered tothe T2-weighted FLAIR scan and downsampled to 096 times096 times 300mm3 The reference standard is therefore onlyavailable for this resolution The decreased performance ofthe FreeSurfer GM segmentation as compared to the otherfreeware packages might be due to the fact that we evaluateon the thick slice T1 sequence instead of the high resolutionT1 Performing the manual segmentations to provide thereference standard is very laborsome and time consumingInstead of letting multiple observers manually segment theMRI datasets or letting one observer manually segmentthe MRI datasets twice much time and effort was spenton creating one reference standard that was as accurate aspossible Therefore we were not able to determine the inter-or intraobserver variability Finally we acknowledge that ourevaluation framework is limited to evaluating the accuracyand robustness over 15 datasets for segmenting GM WMand CSF on 3T MRI scans acquired on a Philips scanner ofa specific group of elderly subjects Many factors influencesegmentation algorithm performance such as the type ofscanner (vendor field strength) the acquisition protocolthe available MRI sequences and the type of subjects

Participating algorithmsmight have been designed for differ-ent types of MRI scans Therefore the five provided trainingdatasets are important for participants to be able to traintheir algorithms on the provided data Some algorithms aredesigned to segment only some components such as onlyGM and WM instead of all three components and usefreely available software such as the brain extraction tool[57] to segment the outer border of the CSF (intracranialvolume) We have chosen to base the final ranking on allthree components but it is therefore important to assess notonly the final ranking but the performance of the individualcomponents as well

Despite these limitations the MRBrainS evaluationframework provides an objective and direct comparison ofsegmentation algorithms The reference standard of the testdata is unknown to the participants the same evaluationmeasures are used for all evaluated algorithms and partici-pants apply their own algorithms to the provided data

In comparison to the online validation engine proposedby Shattuck et al [58] the MRBrainS evaluation frameworkuses 3T MRI data instead of 15T MRI data and evaluates notonly brain versus nonbrain segmentation but also segmenta-tion of gray matter white matter cerebrospinal fluid brainand intracranial volume The availability of many differenttypes of evaluation frameworks will aid in the developmentof more generic and robust algorithms For example inthe NEATBrainS (httpneatbrains15isiuunl) challengeresearchers were challenged to apply their algorithms to data

Computational Intelligence and Neuroscience 9

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

65

70

75

80

85

GM Dice

Dic

e (

)

(a)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

55

60

65

70

75

80

85

90

WM Dice

Dic

e (

)

(b)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

55

60

65

70

75

80

85CSF Dice

Dic

e (

)

(c)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

80

85

90

95

Brain (WM + GM) Dice

Dic

e (

)

(d)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

80

85

90

95

Dic

e (

)

Intracranial volume (WM + GM + CSF) Dice

(e)

Figure 2 Boxplots presenting the evaluation results for the Dice coefficient (1) of the gray matter (GM) white matter (WM) cerebrospinalfluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all 15 test datasets

10 Computational Intelligence and Neuroscience

2468

10121416

GM H-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(a)

2468

10121416

WMH-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(b)

5

10

15

20CSF H-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(c)

5

10

15

20H

-95

(mm

)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Brain (WM + GM) H-95

(d)

5

10

15

20

25

30

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Intracranial volume (WM + GM + CSF) H-95

(e)

Figure 3 Boxplots presenting the evaluation results for the 95th-percentile Hausdorff distance (3) of the gray matter (GM) white matter(WM) cerebrospinal fluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all15 test datasets

Computational Intelligence and Neuroscience 11

0

10

20

30

GM AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(a)

0

10

20

30

40WM AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(b)

0

20

40

60

80

CSF AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(c)

0

5

10

15

20

25

30AV

D (

)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Brain (WM + GM) AVD

(d)

0

5

10

15

20

25

30

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Intracranial volume (WM + GM + CSF) AVD

(e)

Figure 4 Boxplots presenting the evaluation results for the absolute volume difference (4) of the gray matter (GM) white matter (WM)cerebrospinal fluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all 15 testdatasets

12 Computational Intelligence and Neuroscience

T2

UofL

N

FS

S2

‐FLAIR

BioImaging

Narsil

L‐Seg

2_QM

T1‐IR

CMIV

SPM12_T1

SPM12_T1_

LNMBr

_FLAIR

_IR_FLAIR

ains

T1

UB VPML Me

SPM12_T1_ I

FSL‐PVSeg

R

d B

R

F

eference

igr_neuro

MNAB

reeSurfer

BIG

Rob

SPM1

Jedi Mi

R2

arts

2_T1

nd Meld

Figure 5 Illustration of the segmentation results at the height of the basal ganglia (test subject 9 slice 22) The basal ganglia should besegmented as gray matter The first three images show the three MRI sequences The fourth image (reference) is the manually segmentedreference standard (yellow white matter light blue gray matter and dark blue cerebrospinal fluid) The results of the participatingsegmentation algorithms are ordered from the best overall performance (BIGR2) to the worst overall performance (LNMBrains) Majordifferences are present in the segmentation of the basal ganglia and the outer border of the cerebrospinal fluid The arrows indicate examplelocations where the segmentation results differ from the ground truth

Computational Intelligence and Neuroscience 13

T2

UofL BioIm

Nars

FSL‐S

S2_Q

‐FLAIR

aging (860)

il (892)

eg (453)

M (565)

T1‐IR

CMIV (90

SPM12_T1_F

SPM12_T1_IR

LNMBrains

5) UB

(831) SP

_F (815) F

(286)

T1

VPML Med (9

M12_T1_IR (60

SL‐PVSeg (525

R

10) Bigr_

5) MN

) FreeS

eference

neuro (896)

AB (889)

urfer (846)

BIGR2

Robarts

SPM12_T

Jedi Mind M

(852)

(810)

1 (683)

eld (769)

Figure 6 Illustration of the segmentation results in the presence of whitematter lesions (test subject 3 slice 31)Whitematter lesions (WMLs)should be labeled asWM the sensitivity (percentage ofWML voxels is labeled asWM) over all 15 datasets is presented between brackets afterthe team nameThe first three images show the threeMRI sequences T2-weighted fluid attenuated inversion recovery T1-weighted inversionrecovery and T1-weighted scanThe fourth image (reference) is themanually segmented reference standard (red whitematter lesions yellowwhite matter light blue gray matter and dark blue cerebrospinal fluid)The results of the segmentation algorithms are ordered from the bestoverall performance (BIGR2) to the worst overall performance (LNMBrains)The arrows indicate example locations where the segmentationresults differ from the ground truth

14 Computational Intelligence and Neuroscience

fromboth theMRBrainS and theNeoBrainS [59] (brain tissuesegmentation in neonates) challenge Two methods [60ndash62]specifically designed for neonatal brain tissue segmentationshowed a high performance for tissue segmentation on theMRBrainS data Applying algorithms to different types of datahas the potential to lead to new insights and more robustalgorithms The MRBrainS evaluation framework remainsopen for new contributions At the time of writing 21teams had submitted their results on the MRBrainS website(httpmrbrains13isiuunlresultsphp)

5 Conclusion

The MRBrainS challenge online evaluation framework pro-vides direct and objective comparison of automatic andsemiautomatic methods to segment GM WM CSF brainand ICV on 3T multisequence MRI data The first elevenparticipating methods are evaluated and presented in thispaper as well as three commonly used freeware packages(FreeSurfer FSL and SPM12) They provide a benchmarkfor future contributions to the framework The final rankingprovides a quick insight into the overall performance of theevaluated methods in comparison to each other whereasthe individual evaluation measures (Dice 95th-percentileHausdorff distance and absolute volume difference) percomponent (GM WM CSF brain and ICV) can aid inselecting the best method for a specific segmentation goal

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was financially supported by IMDI Grant104002002 (Brainbox) fromZonMw the Netherlands Organ-isation for Health Research and Development within kindsponsoring by Philips theUniversityMedical CenterUtrechtand Eindhoven University of TechnologyThe authors wouldlike to acknowledge the following members of the UtrechtVascular Cognitive Impairment Study Group who were notincluded as coauthors of this paper but were involved in therecruitment of study participants and MRI acquisition at theUMC Utrecht (in alphabetical order by department) E vanden Berg M Brundel S Heringa and L J Kappelle of theDepartment of Neurology P R Luijten and W P Th MMali of the Department of Radiology and A Algra and GE H M Rutten of the Julius Center for Health Sciences andPrimary CareThe research of Geert Jan Biessels and the VCIgroupwas financially supported byVIDIGrant 91711384 fromZonMw and by Grant 2010T073 of the Netherlands HeartFoundation The research of Jeroen de Bresser is financiallysupported by a research talent fellowship of the Univer-sity Medical Center Utrecht (Netherlands) The research ofAnnegreet vanOpbroek andMarleen de Bruijne is financiallysupported by a research grant from NWO (the NetherlandsOrganisation for Scientific Research) The authors would

like to acknowledge MeVis Medical Solutions AG (BremenGermany) for providing MeVisLab Duygu Sarikaya andLiang Zhao acknowledge their Advisor Professor JasonCorsofor his guidance Duygu Sarikaya is supported by NIH 1 R21CA160825-01 and Liang Zhao is partially supported by theChina Scholarship Council (CSC)

References

[1] I Driscoll C Davatzikos Y An et al ldquoLongitudinal pattern ofregional brain volume change differentiates normal aging fromMCIrdquo Neurology vol 72 no 22 pp 1906ndash1913 2009

[2] K Franke and C Gaser ldquoLongitudinal changes in individualBrainAGE in healthy aging mild cognitive impairment andAlzheimerrsquos diseaserdquoGeroPsych vol 25 no 4 pp 235ndash245 2012

[3] MA IkramH A VroomanMWVernooij et al ldquoBrain tissuevolumes in the general elderly populationThe Rotterdam ScanStudyrdquo Neurobiology of Aging vol 29 no 6 pp 882ndash890 2008

[4] A Giorgio and N de Stefano ldquoClinical use of brain volumetryrdquoJournal of Magnetic Resonance Imaging vol 37 no 1 pp 1ndash142013

[5] M W Vannier R L Butterfield D Jordan W A Murphy RG Levitt and M Gado ldquoMultispectral analysis of magneticresonance imagesrdquo Radiology vol 154 no 1 pp 221ndash224 1985

[6] M A Balafar A R Ramli M I Saripan and S MashohorldquoReview of brain MRI image segmentation methodsrdquo ArtificialIntelligence Review vol 33 no 3 pp 261ndash274 2010

[7] J C Bezdek L O Hall and L P Clarke ldquoReview of MR imagesegmentation techniques using pattern recognitionrdquo MedicalPhysics vol 20 no 4 pp 1033ndash1048 1993

[8] L P Clarke R P Velthuizen M A Camacho et al ldquoMRIsegmentation methods and applicationsrdquo Magnetic ResonanceImaging vol 13 no 3 pp 343ndash368 1995

[9] J S Duncan X Papademetris J Yang M Jackowski X Zengand L H Staib ldquoGeometric strategies for neuroanatomic ana-lysis from MRIrdquo NeuroImage vol 23 supplement 1 pp S34ndashS45 2004

[10] D L PhamC Xu and J L Prince ldquoCurrentmethods inmedicalimage segmentationrdquoAnnual Review of Biomedical Engineeringvol 2 no 2000 pp 315ndash337 2000

[11] N Saeed ldquoMagnetic resonance image segmentation usingpattern recognition and applied to image registration andquantitationrdquo NMR in Biomedicine vol 11 no 4-5 pp 157ndash1671998

[12] J S Suri S Singh and L Reden ldquoComputer vision andpattern recognition techniques for 2-D and 3-D MR cerebralcortical segmentation (Part I) a state-of-the-art reviewrdquo PatternAnalysis amp Applications vol 5 no 1 pp 46ndash76 2002

[13] A P Zijdenbos and B M Dawant ldquoBrain segmentation andwhite matter lesion detection in MR imagesrdquo Critical Reviewsin Biomedical Engineering vol 22 no 5-6 pp 401ndash465 1994

[14] K Price ldquoAnything you can do I can do better (No you canrsquot)rdquoComputer Vision Graphics and Image Processing vol 36 no 2-3pp 387ndash391 1986

[15] R de Boer H A Vrooman M A Ikram et al ldquoAccuracy andreproducibility study of automatic MRI brain tissue segmenta-tion methodsrdquo NeuroImage vol 51 no 3 pp 1047ndash1056 2010

[16] J de Bresser M P Portegies A Leemans G J Biessels L JKappelle andM A Viergever ldquoA comparison ofMR based seg-mentation methods for measuring brain atrophy progressionrdquoNeuroImage vol 54 no 2 pp 760ndash768 2011

Computational Intelligence and Neuroscience 15

[17] M B Cuadra L Cammoun T Butz O Cuisenaire and J-P Thiran ldquoComparison and validation of tissue modelizationand statistical classification methods in T1-weighted MR brainimagesrdquo IEEE Transactions on Medical Imaging vol 24 no 12pp 1548ndash1565 2005

[18] F Klauschen A Goldman V Barra A Meyer-Lindenbergand A Lundervold ldquoEvaluation of automated brain MR imagesegmentation and volumetrymethodsrdquoHuman BrainMappingvol 30 no 4 pp 1310ndash1327 2009

[19] H Zaidi T Ruest F Schoenahl and M-L Montandon ldquoCom-parative assessment of statistical brain MR image segmentationalgorithms and their impact on partial volume correction inPETrdquo NeuroImage vol 32 no 4 pp 1591ndash1607 2006

[20] J A Frazier S M Hodge J L Breeze et al ldquoDiagnostic and sexeffects on limbic volumes in early-onset bipolar disorder andschizophreniardquo Schizophrenia Bulletin vol 34 no 1 pp 37ndash462008

[21] A Klein and J Tourville ldquo101 labeled brain images and aconsistent human cortical labeling protocolrdquo Frontiers in Neu-roscience vol 6 article 171 2012

[22] B Van Ginneken T Heimann and M Styner ldquo3D segmen-tation in the clinic a grand challengerdquo in Proceedings of the10th International Conference on Medical Image Computingand Computer Assisted Intervention (MICCAI rsquo07) pp 7ndash15Brisbane Australia October-November 2007

[23] A M Dale B Fischl and M I Sereno ldquoCortical surface-based analysis I Segmentation and surface reconstructionrdquoNeuroImage vol 9 no 2 pp 179ndash194 1999

[24] B Fischl M I Sereno and A M Dale ldquoCortical surface-basedanalysis II inflation flattening and a surface-based coordinatesystemrdquo NeuroImage vol 9 no 2 pp 195ndash207 1999

[25] M Jenkinson C F Beckmann T E J BehrensMWWoolrichand S M Smith ldquoFSLrdquoNeuroImage vol 62 no 2 pp 782ndash7902012

[26] J Ashburner and K J Friston ldquoUnified segmentationrdquo Neu-roImage vol 26 no 3 pp 839ndash851 2005

[27] Y D Reijmer A Leemans M Brundel L J Kappelle and GJ Biessels ldquoDisruption of the cerebral white matter network isrelated to slowing of information processing speed in patientswith type 2 diabetesrdquoDiabetes vol 62 no 6 pp 2112ndash2115 2013

[28] S Klein M Staring K Murphy M A Viergever and J P WPluim ldquoElastix a toolbox for intensity-based medical imageregistrationrdquo IEEE Transactions onMedical Imaging vol 29 no1 pp 196ndash205 2010

[29] K J Friston J T Ashburner S J Kiebel T E Nichols and WD Penny Eds Statistical Parametric Mapping The Analysis ofFunctional Brain Images Academic Press 2011

[30] F Ritter T Boskamp A Homeyer et al ldquoMedical imageanalysis a visual approachrdquo IEEE Pulse vol 2 no 6 pp 60ndash702011

[31] L R Dice ldquoMeasures of the amount of ecologic associationbetween speciesrdquo Ecology vol 26 no 3 pp 297ndash302 1945

[32] D P Huttenlocher G A Klanderman and W J RucklidgeldquoComparing images using the Hausdorff distancerdquo IEEE Trans-actions on Pattern Analysis and Machine Intelligence vol 15 no9 pp 850ndash863 1993

[33] A Alansary A Soliman F Khalifa et al ldquoMAP-based frame-work for segmentation of MR brain images based on visualappearance and prior shaperdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal September 2013

[34] A Jog S Roy J L Prince and A Carass ldquoMR brain segmenta-tion using decision treesrdquo in Proceedings of the MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal Nagoya Japan September 2013

[35] R Katyal S Paneri and M Kuse ldquoGaussian intensity modelwith neighborhood cues for fluid-tissue categorization ofmulti-sequence MR brain imagesrdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal Nagoya Japan September 2013

[36] QMahmoodMAlipoor A Chodorowski AMehnert andMPersson ldquoMultimodal MR brain segmentation using Bayesian-based adaptive mean-shift (BAMS)rdquo in Proceedings of theMICCAI WorkshopsmdashThe MICCAI Grand Challenge on MRBrain Image Segmentation (MRBrainS rsquo13) The Midas JournalSeptember 2013

[37] A van Opbroek F van der Lijn andM de Bruijne ldquoAutomatedbrain-tissue segmentation by multi-feature SVM classificationrdquoin Proceedings of the MICCAI WorkshopsmdashThe MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal September 2013

[38] S Pereira J Festa J A Mariz N Sousa and C A SilvaldquoAutomatic brain tissue segmentation of multi-sequence MRimages using random decision forestsrdquo in Proceedings of theMICCAI Grand Challenge on MR Brain Image Segmentation(MRBrainS rsquo13) The Midas Journal Nagoya Japan September2013

[39] MRajchl J Baxter J Yuan T Peters andAKhan ldquoMulti-atlas-based segmentation with hierarchical max-flowrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[40] D Sarikaya L Zhao and J J Corso ldquoMulti-atlas brain MRIsegmentation with multiway cutrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[41] H Vrooman F van der Lijn and W Niessen ldquoAuto-kNNbrain tissue segmentation using automatically trained k-nearest-neighbor classificationrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[42] S Vyas P Burlina D Kleissas and R Mukherjee ldquoAutomatedwalks using machine learning for segmentationrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[43] C Wang and O Smedby ldquoFully automatic brain segmentationusing model-guided level sets and skeleton-based modelsrdquo inProceedings of theMICCAI Grand Challenge onMRBrain ImageSegmentation (MRBrainS rsquo13) The Midas Journal NagoyaJapan September 2013

[44] C Cortes and V Vapnik ldquoSupport-vector networksrdquo MachineLearning vol 20 no 3 pp 273ndash297 1995

[45] H A Vrooman C A Cocosco F van der Lijn et al ldquoMulti-spectral brain tissue segmentation using automatically trainedk-Nearest-Neighbor classificationrdquo NeuroImage vol 37 no 1pp 71ndash81 2007

[46] C Wang H Frimmel and O Smedby ldquoFast level-set basedimage segmentation using coherent propagationrdquo MedicalPhysics vol 41 no 7 Article ID 073501 2014

16 Computational Intelligence and Neuroscience

[47] A Jog S Roy A Carass and J L Prince ldquoMagnetic resonanceimage synthesis through patch regressionrdquo in Proceedings of theIEEE 10th International Symposium onBiomedical Imaging (ISBIrsquo13) pp 350ndash353 April 2013

[48] J G Sied A P Zijdenbos and A C Evans ldquoA non-parametricmethod for automatic correction of intensity non-uniformity inMRI datardquo IEEE Transactions on Medical Imaging vol 17 no 1pp 87ndash97 1998

[49] A Carass J Cuzzocreo M B Wheeler P-L Bazin S MResnick and J L Prince ldquoSimple paradigm for extra-cerebraltissue removal algorithm and analysisrdquoNeuroImage vol 56 no4 pp 1982ndash1992 2011

[50] M Modat G R Ridgway Z A Taylor et al ldquoFast free-form deformation using graphics processing unitsrdquo ComputerMethods and Programs in Biomedicine vol 98 no 3 pp 278ndash284 2010

[51] M Rajchl J Yuan J A White et al ldquoInteractive hierarchical-flow segmentation of scar tissue from late-enhancement cardiacmr imagesrdquo IEEE Transactions on Medical Imaging vol 33 no1 pp 159ndash172 2014

[52] QMahmood A Chodorowski AMehnert andM Persson ldquoAnovel Bayesian approach to adaptive mean shift segmentationof brain imagesrdquo in Proceedings of the 25th IEEE InternationalSymposiumonComputer-BasedMedical Systems (CBMS rsquo12) pp1ndash6 June 2012

[53] A El-Baz M Nitzken A Alansary A Soliman and MCasanova ldquoBrain Segmentation Method for Young Childrenand Adultsrdquo US Provisional Patent Application 13100

[54] A Alansary A Soliman M Nitzken et al ldquoAn integratedgeometrical and stochastic approach for accurate infant brainextractionrdquo in Proceedings of the IEEE International Conferenceon Image Processing (ICIP rsquo14) pp 3542ndash3546 October 2014

[55] V Popescu M Battaglini W S Hoogstrate et al ldquoOptimizingparameter choice for FSL-Brain Extraction Tool (BET) on 3DT1 images in multiple sclerosisrdquo NeuroImage vol 61 no 4 pp1484ndash1494 2012

[56] J L Whitwell W R Crum H C Watt and N C FoxldquoNormalization of cerebral volumes by use of intracranial vol-ume implications for longitudinal quantitative MR imagingrdquoAmerican Journal of Neuroradiology vol 22 no 8 pp 1483ndash1489 2001

[57] S M Smith ldquoFast robust automated brain extractionrdquo HumanBrain Mapping vol 17 no 3 pp 143ndash155 2002

[58] D W Shattuck G Prasad M Mirza K L Narr and A WToga ldquoOnline resource for validation of brain segmentationmethodsrdquo NeuroImage vol 45 no 2 pp 431ndash439 2009

[59] I Isgum M J N L Benders B Avants et al ldquoEvaluation ofautomatic neonatal brain segmentation algorithms the Neo-BrainS12 challengerdquo Medical Image Analysis vol 20 no 1 pp135ndash151 2015

[60] P Moeskops M J N L Benders S M Chita et al ldquoAutomaticsegmentation of MR brain images of preterm infants usingsupervised classificationrdquo NeuroImage 2015

[61] P Moeskops M A Viergever M J N L Benders and IIsgum ldquoEvaluation of an automatic brain segmentationmethoddeveloped for neonates on adult MR brain imagesrdquo in MedicalImaging Image Processing vol 9413 of Proceedings of SPIEInternational Society for Optics and Photonics Orlando FlaUSA February 2015

[62] L Wang Y Gao F Shi et al ldquoLINKS learning-based multi-source IntegratioN frameworK for Segmentation of infant brainimagesrdquo NeuroImage vol 108 pp 160ndash172 2015

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience 3

cohort study of older (65ndash80 years of age) functionallyindependent individuals without a history of invalidatingstroke or other brain diseases [27] This study was approvedby the local ethics committee of the University MedicalCenter Utrecht (Netherlands) and all participants signed aninformed consent form To be able to test the robustnessof the segmentation algorithms in the context of ageing-related pathology the subjects were selected to have varyingdegrees of atrophy and white matter lesions Scans withmajor artefacts were excluded MRI scans were acquired ona 30 T Philips AchievaMR scanner at the University MedicalCenter Utrecht (Netherlands) The following sequences wereacquired and used for the evaluation framework 3D T1 (TR79ms TE 45ms) T1-IR (TR 4416ms TE 15ms and TI400ms) and T2- FLAIR (TR 11000ms TE 125ms and TI2800ms) Since the focus of the MRBrainS evaluation frame-work is on comparing different segmentation algorithms weperformed two preprocessing steps to limit the influenceof different registration and bias correction algorithms onthe segmentation results The sequences were aligned byrigid registration using Elastix [28] and bias correction wasperformed using SPM8 [29] After registration the voxel sizewithin all provided sequences (T1 T1 IR and T2 FLAIR)was 096 times 096 times 300mm3 The original 3D T1 sequence(voxel size 10 times 10 times 10mm3) was provided as well Fivedatasets that were representative for the overall data (2 male3 female varying degrees of atrophy andwhitematter lesions)were selected for training The remaining fifteen datasets areprovided as test data

212 Reference Standard Manual segmentations were per-formed to obtain a reference standard for the evaluationframework All axial slices of the 20 datasets (096 times 096times 300mm3) were manually segmented by trained researchassistants in a darkened room with optimal viewing con-ditions All segmentations were checked and corrected bythree experts a neurologist in training a neuroradiologistin training and a medical image processing scientist Toperform the manual segmentations an in-house developedtool based on MeVisLab (MeVis Medical Solutions AGBremen Germany) was used employing a freehand splinedrawing technique [30] The closed freehand spline drawingtechnique was used to delineate the outline of each brainstructure starting at the innermost structures (Figure 1(a))working outward The closed contours were converted tohard segmentations and the inner structures were itera-tively subtracted from the outer structures to construct thefinal hard segmentation image (Figure 1(b)) The followingstructures were segmented and are available for trainingcortical gray matter (1) basal ganglia (2) white matter (3)white matter lesions (4) peripheral cerebrospinal fluid (5)lateral ventricles (6) cerebellum (7) and brainstem (8)Thesestructures can be merged into gray matter (1 2) white matter(3 4) and cerebrospinal fluid (5 6) The cerebellum andbrainstem are excluded from the evaluation All structureswere segmented on theT1-weighted scans thatwere registeredto the FLAIR scans except for the white matter lesions(WMLs) and the CSF outer border (used to determine

the intracranial volume) The WMLs were segmented onthe FLAIR scan by the neurologist in training and checkedand corrected by the neuroradiologist in training The CSFouter border was segmented using both the T1-weightedand the T1-weighted IR scan since the T1-weighted IR scanshows higher contrast at the borders of the intracranialvolumeTheCSF segmentation includes all vessels (includingthe superior sagittal sinus and the transverse sinuses) andnonbrain structures such as the cerebral falx and choroidplexuses

213 Evaluation To evaluate the segmentation results weuse three types of measures a spatial overlap measure aboundary distance measure and a volumetric measure TheDice [31] coefficient is used to determine the spatial overlapand is defined as

119863 =2 |119860 cap 119866|

|119860| + |119866|

sdot 100 (1)

where 119860 is the segmentation result 119866 is the referencestandard and 119863 is the Dice expressed as percentages The95th-percentile of theHausdorffdistance is used to determinethe distance between the segmentation boundaries Theconventional Hausdorff distance uses the maximum whichis very sensitive to outliers To correct for outliers we usethe 95th-percentile of theHausdorff distance by selecting the119870th ranked distance as proposed by Huttenlocher et al [32]

ℎ95 (119860 119866) =

95119870

th119886isin119860

min119892isin119866

1003817100381710038171003817119892 minus 1198861003817100381710038171003817 (2)

where 95119870th119886isin119860

is the119870th rankedminimumEuclidean distancewith 119870119873

119886= 95 119860 is the set of boundary points

1198861 119886

119873119886

of the segmentation result and 119866 is the set ofboundary points 119892

1 119892

119873119892

of the reference standard The95th-percentile of the Hausdorff distance is defined as

11986795(119860 119866) = max (ℎ

95(119860 119866) ℎ

95(119866 119860)) (3)

The third measure is the percentage absolute volume differ-ence defined as

AVD =10038161003816100381610038161003816119881119886minus 119881119892

10038161003816100381610038161003816

119881119892

sdot 100 (4)

where 119881119886is the volume of the segmentation result and 119881

119892

is the volume of the reference standard These measures areused to evaluate the following brain structures in each of thefifteen test datasets GM WM CSF brain (GM + WM) andintracranial volume (GM + WM + CSF) The brainstem andcerebellum are excluded from the evaluation

214 Ranking To compare the segmentation algorithmsthat participate in the MRBrainS evaluation framework thealgorithms are ranked based on their overall performance tosegment GMWM and CSF Each of these components (119862 =GMWMCSF) is evaluated by using the three evaluationmeasures (119872 = 119863119867

95AVD) described in Section 213

4 Computational Intelligence and Neuroscience

(a) (b) (c) (d) (e)

Figure 1 Example of themanually drawn contours (a) the resulting hard segmentationmap (GM light blueWM yellow andCSF dark blue)that is used as the reference standard (b) the T1-weighted scan (c) the T1-weighted inversion recovery (IR) scan (d) and the T2-weightedfluid attenuated inversion recovery (FLAIR) scan (e)

For each component 119888 isin 119862 and each evaluation measure119898 isin119872 the mean and standard deviation are determined over all15 test datasets The segmentation algorithms are then sortedon the mean 119863 value in descending order and on the mean11986795

and AVD value in ascending order Each segmentationalgorithm receives a rank (119903) between 1 (ranked best) and 119899(number of participating algorithms) for each component 119888and each evaluation measure 119898 The final ranking is basedon the overall score of each algorithm which is the sum overall ranks defined as

119904 =

|119872|

sum

119898=0

|119862|

sum

119888=0

119903119898119888 (5)

where 119903119898119888

is the rank of the segmentation algorithm formeasure 119898 of component 119888 For the final ranking 119903 theoverall scores 119904 are sorted in ascending order and rankedfrom 1 to 119899 In case two or more algorithms have equalscores the standard deviation over all 15 test datasets is takeninto account to determine the final rank The segmentationalgorithms are then sorted on the standard deviation inascending order and ranked for each component 119888 and eachevaluation measure119898 The overall score is determined using(5) and the algorithms are sorted based on this score inascending order and ranked from 1 to 119899 The algorithms thathave equal overall scores based on the mean value are thenranked based on this standard deviation rank

22 EvaluatedMethods Theevaluation framework describedin Section 21 was launched at the MRBrainS13 challengeworkshop at the Medical Image Computing and ComputerAssisted Intervention (MICCAI) conference on September26th in 2013 For the workshop challenge the test datasetswere split into twelve off-site and three on-site test datasetsFor the off-site part teams could register on the MRBrainSwebsite (httpmrbrains13isiuunl) and download the fivetraining and twelve test datasets A time slot of eight weekswas available for teams to download the data train theiralgorithms segment the test datasets and submit their resultson the website Fifty-eight teams downloaded the data ofwhich twelve submitted their segmentation results The eval-uation results were reported to the twelve teams and all teamssubmitted a workshop paper to the MRBrainS13 challengeworkshop at MICCAI Eleven teams presented their results

at the workshop and segmented the three on-site test datasetslive at the workshop within a time slot of 35 hours Thesealgorithms provide a benchmark for the proposed evaluationframework and are briefly described in Sections 221ndash2211in alphabetical order of teamsrsquo names The teamsrsquo namesare used in the paper to identify the methods For a fulldescription of the methods we refer to the workshop papers[33ndash43] In Section 2212 we describe the evaluated freewarepackages

221 BIGR2 [37] This multifeature SVM [37] method clas-sifies voxels by using a Support Vector Machine (SVM)classifier [44] with a Gaussian kernel Besides spatial featuresand intensity information from all three MRI sequences theSVM classifier incorporates Gaussian-scale-space features tofacilitate a smooth segmentation Skull stripping is performedby nonrigid registration of the masks of the training imagesto the target image

222 Bigr neuro [41] This auto-kNN [45] method is basedon an automatically trained kNN-classifier First a proba-bilistic tissue atlas is generated by nonrigidly registering themanually annotated atlases to the subject of interest Trainingsamples are obtained by thresholding the probabilistic atlasand subsequently pruning the feature space White matterlesions are detected by applying an adaptive threshold deter-mined from the tissue segmentation to the FLAIR sequence

223 CMIV [43] A statistical-model-guided level-setmethod is used to segment the skull brain ventricles andbasal ganglia Then a skeleton-based model is created byextracting the midsurface of the gray matter and definingthe thickness This model is incorporated into a level-setframework to guide the cortical gray matter segmentationThe coherent propagation algorithm [46] is used to acceleratethe level-set evolution

224 Jedi MindMeld [42] Thismethod starts by preprocess-ing the data via anisotropic diffusion For each 2D slice of alabeled dataset the canny edge pixels are extracted and theTourist Walk is computed This is done for axial sagittal andcoronal views Machine learning is used with these featuresto automatically label edge pixels in an unlabeled dataset

Computational Intelligence and Neuroscience 5

Finally these labels are used by the Random Walker forautomatic segmentation

225 LNMBrains [35] The voxel intensities of all MRIsequences are modelled as a Gaussian distribution for eachlabel The parameters of the Gaussian distributions areevaluated asmaximum likelihood estimates and the posteriorprobability of each label is determined by using Bayesianestimation A feature set consisting of regional intensitytexture spatial location of voxels and the posterior prob-ability estimates is used to classify each voxel into CSFWM GM or background by using a multicategory SVMclassifier

226 MNAB [38] This method uses Random DecisionForests to classify the voxels into GM WM and CSF Itstarts by a skull stripping procedure followed by an intensitynormalization of each MRI sequence Feature extraction isthen performed on the intensities posterior probabilitiesneighborhood statistics tissue atlases and gradient mag-nitude After classification isolated voxels are removed bypostprocessing

227 Narsil [34] This is a model-free algorithm that usesensembles of decision trees [47] to learn the mapping fromimage features to the corresponding tissue label The ensem-bles of decision trees are constructed from correspondingimage patches of the provided T1 and FLAIR scans withmanual segmentations The N3 algorithm [48] was used foradditional inhomogeneity correction and SPECTRE [49] wasused for skull stripping

228 Robarts [39] Multiatlas registration [50] with the T1training images was used to propagate labels to generatesample histograms in a log-likelihood intensity model andprobabilistic shape priors These were employed in a MAPdata term and regularized via computation of a hierar-chical max-flow [51] A brain mask from registration ofthe T1-IR training images was used to obtain the finalresults

229 S2 QM [36] This method [52] is based on Bayesian-based adaptive mean shift and the voxel-weighted 119870-meansalgorithm The former is used to segment the brain into alarge number of clusters or modes The latter is employed toassign these clusters to one of the three components WMGM or CSF

2210 UB VPMLMed [40] This method creates a multiatlasby registering the training images to the subject imageand then propagating the corresponding labels to a fullyconnected graph on the subject image Label fusion thencombines the multiple labels into one label at each voxelwith intensity similarity based weighted voting Finally themethod clusters the graph using multiway cut in order toachieve the final segmentation

2211 UofL BioImaging [33] This is an automated MAP-based method aimed at unsupervised segmentation of differ-ent brain tissues from T1-weighted MRI It is based on theintegration of a probabilistic shape prior a first-order inten-sity model using a Linear Combination of Discrete Gaussians(LCDG) and a second-order appearance model These threefeatures are integrated into a two-level joint Markov-GibbsRandom Field (MGRF) model of T1-MR brain imagesSkull stripping was performed using BET2 [40] followed byan adaptive threshold-based technique to restore the outerborder of the CSF using both T1 andT1-IR this techniquewasnot described in [33] due to a US patent application [53] butis described in [54] This method was applied semiautomat-ically to the MRBrainS test data due to per scan parametertuning

2212 Freeware Packages Next to the methods evaluatedat the workshop we evaluated three commonly used free-ware packages for MR brain image segmentation FreeSurfer(httpsurfernmrmghharvardedu) [23 24] FSL (httpfslfmriboxacukfslfslwiki) [25] and SPM12 (httpwwwfilionuclacukspm) [26] All packages were applied usingthe default settings unless mentioned otherwise FreeSurfer(v530) was applied to the high resolution T1 sequence Themri label2vol tool was used to map the labels on the thickslice T1 that was used for the evaluation FSL (v50) wasdirectly applied to the thick slice T1 and provides both apveseg and a seg file as binary output We evaluated bothof these files The fractional intensity threshold parameterldquo119891rdquo of the BET tool that sets the brainnonbrain intensitythreshold was set according to [55] at 02 (Philips Achieva3T setting) SPM12 was directly applied to the thick sliceT1 sequence as well However it also provides the optionto add multiple MRI sequences Therefore we evaluatedSPM12 not only on the thick slice T1 sequence but addedthe T1-IR and the T2-FLAIR scan as well and tested variouscombinationsThe number of Gaussians was set according tothe SPM manual to two for GM two for WM and two forCSF

23 Statistical Analysis All evaluated methods werecompared to the reference standard In summary ofthe results the mean and standard deviation overall 15 test datasets were calculated per component(GM WM and CSF) and combination of components(brain intracranial volume) and per evaluation measure(Dice 95th-percentile Hausdorff distance and absolutevolume difference) for each of the evaluated methodsBoxplots were created using R version 303 (R projectfor statistical computing (httpwwwr-projectorg))Since white matter lesions should be segmented as whitematter the percentage of white matter lesion voxelssegmented as white matter (sensitivity) was calculatedfor each algorithm over all 15 test datasets to evaluatethe robustness of the segmentation algorithms againstpathology

6 Computational Intelligence and Neuroscience

3 Results

Table 1 presents the final ranking (119903) of the evaluatedmethodsthat participated in the workshop as well as the evalu-ated freeware packages During the workshop team UofLBioImaging ranked first and BIGR2 ranked second with onepoint difference in the overall score 119904 (5) However addingthe results of the freeware packages resulted in an equal scorefor UofL BioImaging and BIGR2 Therefore the standarddeviation rank was taken into account and BIGR2 is rankedfirst with standard deviation rank four and UofL BioImagingis ranked second with standard deviation rank eight Table 1further presents the mean standard deviation and rank foreach evaluation measure (119863119867

95 and AVD) and component

(GM WM and CSF) as well as the brain (WM + GM) andintracranial volume (WM+GM+CSF) Team BIGR2 scoredbest for theGMWM and brain segmentation and teamUofLBioImaging for the CSF segmentation Team Robarts scoredbest for the intracranial volume segmentation The boxplotsfor all evaluation measures and components are shown inFigures 2ndash4 and include the results of the freeware packagesFigure 5 shows an example of the segmentation results atthe height of the basal ganglia (slice 22 of test subject 9)The sensitivity of the algorithms to segment white matterlesions as WM and examples of the segmentation results inthe presence of white matter lesions (slice 31 of test subject3) are shown in Figure 6 Team UB VPML Med scores thehighest sensitivity of white matter lesions segmented as whitematter and is therefore most robust in the presence of thistype of pathology

4 Discussion

In this paper we proposed the MRBrainS challenge onlineevaluation framework to evaluate automatic and semiauto-matic algorithms for segmenting GM WM and CSF on3T multisequence (T1 T1-IR and T2-FLAIR) MRI scans ofthe brain We have evaluated and presented the results ofeleven segmentation algorithms that provide a benchmarkfor algorithms that will use the online evaluation frameworkto evaluate their performance Team UofL BioImaging andBIGR2 have equal overall scores but BIGR2 was rankedfirst based on the standard deviation ranking The evalu-ated methods represent a wide variety of algorithms thatinclude Markov random field models clustering approachesdeformable models and atlas-based approaches and classi-fiers (SVM KNN and decision trees) The presented evalua-tion framework provides an insight into the performance ofthese algorithms in terms of accuracy and robustness Variousfactors influence the choice for a certain method aboveothers We provide three measures that could aid in selectingthe method that is most appropriate for the segmentationgoal at hand a boundarymeasure (95th-percentile Hausdorffdistance 119867

95) an overlap measure (Dice coefficient 119863) and

a volume measure (absolute volume difference AVD) Allthree measures are taken into account for the final rankingof the methods This ranking was designed to get a quickinsight into how the methods perform in comparison to eachother The best overall method is the method that performs

well for all three measures and all three components (GMWM andCSF) However whichmethod to select depends onthe segmentation goal at hand Not all measures are relevantfor all segmentation goals For example if segmentation isused for brain volumetry [4] the overlap (119863) and volume(AVD) measures of the brain and intracranial volume (usedfor normalization [56]) segmentations are important to takeinto account On the other hand if segmentation is usedfor cortical thickness measurements the focus should be onthe gray matter boundary (119867

95) and overlap (119863) measures

Therefore the final ranking should be used to get a first insightinto the overall performance after which the performanceof the measures and components that are most relevant forthe segmentation goal at hand should be considered Besidesaccuracy robustness could also influence the choice for acertain method above others For example team UB VPMLMed shows a high sensitivity score for segmenting whitemat-ter lesions as white matter (Figure 6) and shows a consistentsegmentation performance of gray and white matter over all15 test datasets (Figures 2ndash4) This could be beneficial forsegmenting scans of populations with white matter lesionsbut is less important if the goal is to segment scans ofyoung healthy subjects In the latter case the most accuratesegmentation for gray and white matter (team BIGR2) ismore interesting If a segmentation algorithm is to be usedin clinical practice speed is an important consideration aswell The runtime of the evaluated methods is reported inTable 1 However these runtimes are merely an indicationof the required time since academic software is generallynot optimized for speed and the runtime is measured ondifferent computers andplatformsAnother relevant aspect ofthe evaluation framework is the comparison of multi- versussingle-sequence approaches For example most methodsstruggle with the segmentation of the intracranial volume onthe T1-weighted scan There is no contrast between the CSFand the skull and the contrast between the dura mater andthe CSF is not always sufficient Team Robarts used an atlas-based registration approach on the T1-IR scan (good contrastbetween skull and CSF) to segment the intracranial volumewhich resulted in the best performance for intracranialvolume segmentation (Table 1 Figures 2ndash4) Most methodsadd the T2-FLAIR scan to improve robustness against whitematter lesions (Table 1 Figure 6) Although using only theT1-weighted scan and incorporating prior shape information(team UofL BioImaging) can be very effective also thefreeware packages support this as well Since FreeSurfer isan atlas-based method it uses prior information and is themost robust of all freeware packages to white matter lesionsHowever adding the T2 FLAIR scan to SPM12 increasesrobustness against white matter lesions as well as comparedto applying SPM12 to the T1 scan only (Figure 6) In generalSPM12with the T1 and the T2-FLAIR sequence performswellin comparison to the other freeware packages (Table 1 andFigures 2ndash4) on the thick slice MRI scans Although addingthe T1-IR scan to SPM increases the performance of the CSFand ICV segmentations as compared to using only the T1 andT2-FLAIR sequence it decreases the performance of the GMand WM segmentations Therefore adding all sequences toSPM12 did not result in a better overall performance

Computational Intelligence and Neuroscience 7

Table 1 Results of the 11 evaluated algorithms presented at the workshop and the evaluated freeware packages on the 15 test datasets Thealgorithms are ranked (119903) based on their overall score (119904) by using (5) This score is based on the ranks of the gray matter (GM) white matter(WM) and cerebrospinal fluid (CSF) segmentation and the three evaluated measures Dice coefficient (119863 in ) 95th-percentile Hausdorffdistance (119867

95in mm) and the absolute volume difference (AVD in ) The rank 119903

119898119888denotes the rank based on the mean (120583) over all 15 test

datasets for each measure 119898 (0 119863 1 11986795 and 2 AVD) and component 119888 (0 GM 1 WM 2 CSF 3 brain (WM + GM) and 4 intracranial

volume (ICV = WM + GM + CSF)) Teams BIGR2 and UofL BioImaging and FreeSurfer and Jedi Mind Meld have equal scores based onthe mean (120583) therefore the ranking based on the standard deviation (120590) is taken into account to determine the final rank (BIGR2 120590 rank 4UofL BioImaging 120590 rank 8 FreeSurfer 120590 rank 13 and Jedi MindMeld 120590 rank 17) Columns 2 and 3 present the average runtime 119905 per scan inseconds (s) minutes (m) or hours (h) and the scans (T1 T1-weighted scan 3D T1 3D T1-weighted scan IR T1-weighted inversion recovery(IR) and F T1-weighted FLAIR) that are used for processing

119903 Team 119905 Scans

GM WM CSF

119904

Brain ICV119903001199031011990320119903011199031111990321119903021199031211990322

119903031199031311990323119903041199031411990324

119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590)

1 BIGR2 35m T1 IR F1 2 4 2 2 4 4 5 14

381 2 12 8 4 12

847 19 61 884 24 60 783 32 23 951 27 32 960 39 52(13) (04) (33) (12) (05) (51) (50) (08) (17) (05) (08) (16) (13) (11) (30)

2 UofLBioImaginglowast 6 s T1

5 1 9 4 1 13 2 2 138

2 1 12 5 2 5830 17 86 879 22 87 789 27 97 949 24 39 967 34 18(15) (03) (54) (20) (06) (66) (42) (05) (10) (06) (05) (20) (08) (06) (20)

3 CMIV 3m T1 F6 7 5 5 3 10 3 3 8

505 7 2 4 3 11

824 27 68 877 24 73 786 30 14 945 38 26 968 38 49(14) (04) (40) (16) (04) (38) (31) (04) (59) (05) (11) (22) (08) (13) (23)

4 UB VPMLMed 30m T1 IR F

4 4 2 1 5 7 8 13 1761

4 4 1 10 11 15833 21 59 886 27 71 748 43 31 946 28 24 948 66 77(13) (03) (53) (17) (04) (38) (71) (17) (19) (06) (04) (18) (20) (20) (41)

5 Bigr neuro 2 h T1 F7 13 3 6 6 9 6 4 10

647 10 10 7 5 8

815 37 59 873 30 73 782 32 16 940 46 36 963 39 35(17) (09) (42) (14) (04) (38) (47) (06) (14) (08) (14) (24) (12) (09) (27)

6 Robarts 16m 3D T1 IR11 3 15 8 8 6 1 1 13

6616 3 18 1 1 1

797 20 98 862 31 71 803 27 20 931 28 79 979 26 09(24) (01) (73) (13) (04) (62) (41) (05) (13) (16) (05) (36) (03) (04) (07)

7 Narsil 2m T1 F3 5 1 7 11 2 17 18 7

713 5 3 16 18 9

835 23 55 871 33 58 666 133 14 948 29 29 925 24 37(18) (04) (44) (13) (09) (53) (24) (54) (95) (05) (05) (20) (05) (89) (17)

8 SPM T1 F 3m T1 F8 9 16 9 7 1 9 14 2

759 14 14 6 14 4

812 29 10 860 30 52 741 46 10 939 58 53 966 82 15(22) (03) (85) (15) (01) (38) (34) (06) (47) (10) (22) (38) (02) (30) (10)

9 SPM T1 IR 3m T1 IR12 11 7 16 12 5 5 10 3

818 11 7 2 8 2

794 30 72 835 36 63 783 40 10 939 46 34 977 65 10(21) (04) (63) (21) (03) (46) (38) (06) (57) (08) (12) (28) (02) (13) (08)

10 MNAB 15m T1 IR F2 8 12 3 4 11 15 15 16

866 9 11 15 12 17

839 28 91 880 27 78 681 49 29 945 45 38 925 71 97(21) (09) (65) (12) (08) (40) (40) (22) (21) (10) (20) (32) (11) (42) (47)

11 SPM T1 3m T19 10 6 11 10 3 11 16 15

9110 8 6 9 13 13

803 30 69 856 31 60 707 53 23 939 44 32 953 81 55(24) (05) (68) (17) (01) (41) (38) (15) (157) (09) (16) (29) (09) (37) (37)

12 FSL Seg 10m T113 16 10 10 13 14 12 6 5

9913 13 4 12 6 6

787 43 86 860 37 115 699 34 12 933 55 30 942 53 34(22) (12) (63) (26) (08) (63) (28) (02) (103) (08) (14) (15) (08) (11) (15)

8 Computational Intelligence and Neuroscience

Table 1 Continued

119903 Team 119905 Scans

GM WM CSF

119904

Brain ICV119903001199031011990320119903011199031111990321119903021199031211990322

119903031199031311990323119903041199031411990324

119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590)

13 SPM T1 IR F 4m T1 IR F10 12 18 15 14 12 7 11 6

10512 15 13 3 15 3

801 30 139 836 38 84 769 41 12 936 59 51 977 82 12(24) (02) (96) (21) (05) (52) (31) (05) (60) (11) (18) (36) (02) (18) (09)

14 FSL PVSeg 10m T115 15 8 13 15 17 13 7 4

10711 16 9 13 7 7

777 43 84 848 38 197 695 34 11 936 61 35 942 53 34(26) (13) (65) (32) (09) (10) (22) (03) (58) (09) (16) (30) (08) (11) (15)

15 FreeSurfer 1 h 3D T116 6 17 12 9 8 18 12 18

11617 6 16 17 10 18

774 23 121 852 31 72 658 43 50 923 36 63 924 66 10(20) (06) (60) (22) (05) (46) (37) (06) (196) (08) (06) (34) (09) (04) (45)

16 Jedi MindMeld 27 s T1 IR F

14 14 11 17 16 15 10 8 11116

15 12 8 11 16 14778 39 89 806 45 137 733 37 18 932 54 35 943 20 68(59) (28) (76) (96) (36) (12) (35) (08) (90) (17) (48) (30) (17) (37) (31)

17 S2 QM 15 h T1 IR F17 17 13 14 17 18 16 9 9

13014 17 15 14 9 10

764 55 93 839 49 248 679 38 14 933 70 55 937 65 40(34) (30) (64) (34) (32) (10) (23) (06) (99) (14) (35) (45) (11) (14) (22)

18 LNMBrains 5m T1 IR F18 18 14 18 18 16 14 17 12

14518 18 17 18 17 16

728 68 95 783 68 153 688 77 20 885 86 74 892 204 79(53) (23) (76) (64) (35) (13) (66) (24) (13) (45) (28) (77) (48) (37) (83)

lowastSemiautomatic due to per scan parameter tuning

Besides the advantages of the MRBrainS evaluationframework there are some limitations that should be takeninto account The T1-weighted IR and the T2-weightedFLAIR scan were acquired with a lower resolution (096times 096 times 300mm3) than the 3D T1-weighted scan (10 times10 times 10mm3) To be able to provide a registered multise-quence dataset the 3D T1-weighted scan was registered tothe T2-weighted FLAIR scan and downsampled to 096 times096 times 300mm3 The reference standard is therefore onlyavailable for this resolution The decreased performance ofthe FreeSurfer GM segmentation as compared to the otherfreeware packages might be due to the fact that we evaluateon the thick slice T1 sequence instead of the high resolutionT1 Performing the manual segmentations to provide thereference standard is very laborsome and time consumingInstead of letting multiple observers manually segment theMRI datasets or letting one observer manually segmentthe MRI datasets twice much time and effort was spenton creating one reference standard that was as accurate aspossible Therefore we were not able to determine the inter-or intraobserver variability Finally we acknowledge that ourevaluation framework is limited to evaluating the accuracyand robustness over 15 datasets for segmenting GM WMand CSF on 3T MRI scans acquired on a Philips scanner ofa specific group of elderly subjects Many factors influencesegmentation algorithm performance such as the type ofscanner (vendor field strength) the acquisition protocolthe available MRI sequences and the type of subjects

Participating algorithmsmight have been designed for differ-ent types of MRI scans Therefore the five provided trainingdatasets are important for participants to be able to traintheir algorithms on the provided data Some algorithms aredesigned to segment only some components such as onlyGM and WM instead of all three components and usefreely available software such as the brain extraction tool[57] to segment the outer border of the CSF (intracranialvolume) We have chosen to base the final ranking on allthree components but it is therefore important to assess notonly the final ranking but the performance of the individualcomponents as well

Despite these limitations the MRBrainS evaluationframework provides an objective and direct comparison ofsegmentation algorithms The reference standard of the testdata is unknown to the participants the same evaluationmeasures are used for all evaluated algorithms and partici-pants apply their own algorithms to the provided data

In comparison to the online validation engine proposedby Shattuck et al [58] the MRBrainS evaluation frameworkuses 3T MRI data instead of 15T MRI data and evaluates notonly brain versus nonbrain segmentation but also segmenta-tion of gray matter white matter cerebrospinal fluid brainand intracranial volume The availability of many differenttypes of evaluation frameworks will aid in the developmentof more generic and robust algorithms For example inthe NEATBrainS (httpneatbrains15isiuunl) challengeresearchers were challenged to apply their algorithms to data

Computational Intelligence and Neuroscience 9

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

65

70

75

80

85

GM Dice

Dic

e (

)

(a)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

55

60

65

70

75

80

85

90

WM Dice

Dic

e (

)

(b)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

55

60

65

70

75

80

85CSF Dice

Dic

e (

)

(c)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

80

85

90

95

Brain (WM + GM) Dice

Dic

e (

)

(d)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

80

85

90

95

Dic

e (

)

Intracranial volume (WM + GM + CSF) Dice

(e)

Figure 2 Boxplots presenting the evaluation results for the Dice coefficient (1) of the gray matter (GM) white matter (WM) cerebrospinalfluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all 15 test datasets

10 Computational Intelligence and Neuroscience

2468

10121416

GM H-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(a)

2468

10121416

WMH-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(b)

5

10

15

20CSF H-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(c)

5

10

15

20H

-95

(mm

)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Brain (WM + GM) H-95

(d)

5

10

15

20

25

30

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Intracranial volume (WM + GM + CSF) H-95

(e)

Figure 3 Boxplots presenting the evaluation results for the 95th-percentile Hausdorff distance (3) of the gray matter (GM) white matter(WM) cerebrospinal fluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all15 test datasets

Computational Intelligence and Neuroscience 11

0

10

20

30

GM AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(a)

0

10

20

30

40WM AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(b)

0

20

40

60

80

CSF AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(c)

0

5

10

15

20

25

30AV

D (

)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Brain (WM + GM) AVD

(d)

0

5

10

15

20

25

30

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Intracranial volume (WM + GM + CSF) AVD

(e)

Figure 4 Boxplots presenting the evaluation results for the absolute volume difference (4) of the gray matter (GM) white matter (WM)cerebrospinal fluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all 15 testdatasets

12 Computational Intelligence and Neuroscience

T2

UofL

N

FS

S2

‐FLAIR

BioImaging

Narsil

L‐Seg

2_QM

T1‐IR

CMIV

SPM12_T1

SPM12_T1_

LNMBr

_FLAIR

_IR_FLAIR

ains

T1

UB VPML Me

SPM12_T1_ I

FSL‐PVSeg

R

d B

R

F

eference

igr_neuro

MNAB

reeSurfer

BIG

Rob

SPM1

Jedi Mi

R2

arts

2_T1

nd Meld

Figure 5 Illustration of the segmentation results at the height of the basal ganglia (test subject 9 slice 22) The basal ganglia should besegmented as gray matter The first three images show the three MRI sequences The fourth image (reference) is the manually segmentedreference standard (yellow white matter light blue gray matter and dark blue cerebrospinal fluid) The results of the participatingsegmentation algorithms are ordered from the best overall performance (BIGR2) to the worst overall performance (LNMBrains) Majordifferences are present in the segmentation of the basal ganglia and the outer border of the cerebrospinal fluid The arrows indicate examplelocations where the segmentation results differ from the ground truth

Computational Intelligence and Neuroscience 13

T2

UofL BioIm

Nars

FSL‐S

S2_Q

‐FLAIR

aging (860)

il (892)

eg (453)

M (565)

T1‐IR

CMIV (90

SPM12_T1_F

SPM12_T1_IR

LNMBrains

5) UB

(831) SP

_F (815) F

(286)

T1

VPML Med (9

M12_T1_IR (60

SL‐PVSeg (525

R

10) Bigr_

5) MN

) FreeS

eference

neuro (896)

AB (889)

urfer (846)

BIGR2

Robarts

SPM12_T

Jedi Mind M

(852)

(810)

1 (683)

eld (769)

Figure 6 Illustration of the segmentation results in the presence of whitematter lesions (test subject 3 slice 31)Whitematter lesions (WMLs)should be labeled asWM the sensitivity (percentage ofWML voxels is labeled asWM) over all 15 datasets is presented between brackets afterthe team nameThe first three images show the threeMRI sequences T2-weighted fluid attenuated inversion recovery T1-weighted inversionrecovery and T1-weighted scanThe fourth image (reference) is themanually segmented reference standard (red whitematter lesions yellowwhite matter light blue gray matter and dark blue cerebrospinal fluid)The results of the segmentation algorithms are ordered from the bestoverall performance (BIGR2) to the worst overall performance (LNMBrains)The arrows indicate example locations where the segmentationresults differ from the ground truth

14 Computational Intelligence and Neuroscience

fromboth theMRBrainS and theNeoBrainS [59] (brain tissuesegmentation in neonates) challenge Two methods [60ndash62]specifically designed for neonatal brain tissue segmentationshowed a high performance for tissue segmentation on theMRBrainS data Applying algorithms to different types of datahas the potential to lead to new insights and more robustalgorithms The MRBrainS evaluation framework remainsopen for new contributions At the time of writing 21teams had submitted their results on the MRBrainS website(httpmrbrains13isiuunlresultsphp)

5 Conclusion

The MRBrainS challenge online evaluation framework pro-vides direct and objective comparison of automatic andsemiautomatic methods to segment GM WM CSF brainand ICV on 3T multisequence MRI data The first elevenparticipating methods are evaluated and presented in thispaper as well as three commonly used freeware packages(FreeSurfer FSL and SPM12) They provide a benchmarkfor future contributions to the framework The final rankingprovides a quick insight into the overall performance of theevaluated methods in comparison to each other whereasthe individual evaluation measures (Dice 95th-percentileHausdorff distance and absolute volume difference) percomponent (GM WM CSF brain and ICV) can aid inselecting the best method for a specific segmentation goal

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was financially supported by IMDI Grant104002002 (Brainbox) fromZonMw the Netherlands Organ-isation for Health Research and Development within kindsponsoring by Philips theUniversityMedical CenterUtrechtand Eindhoven University of TechnologyThe authors wouldlike to acknowledge the following members of the UtrechtVascular Cognitive Impairment Study Group who were notincluded as coauthors of this paper but were involved in therecruitment of study participants and MRI acquisition at theUMC Utrecht (in alphabetical order by department) E vanden Berg M Brundel S Heringa and L J Kappelle of theDepartment of Neurology P R Luijten and W P Th MMali of the Department of Radiology and A Algra and GE H M Rutten of the Julius Center for Health Sciences andPrimary CareThe research of Geert Jan Biessels and the VCIgroupwas financially supported byVIDIGrant 91711384 fromZonMw and by Grant 2010T073 of the Netherlands HeartFoundation The research of Jeroen de Bresser is financiallysupported by a research talent fellowship of the Univer-sity Medical Center Utrecht (Netherlands) The research ofAnnegreet vanOpbroek andMarleen de Bruijne is financiallysupported by a research grant from NWO (the NetherlandsOrganisation for Scientific Research) The authors would

like to acknowledge MeVis Medical Solutions AG (BremenGermany) for providing MeVisLab Duygu Sarikaya andLiang Zhao acknowledge their Advisor Professor JasonCorsofor his guidance Duygu Sarikaya is supported by NIH 1 R21CA160825-01 and Liang Zhao is partially supported by theChina Scholarship Council (CSC)

References

[1] I Driscoll C Davatzikos Y An et al ldquoLongitudinal pattern ofregional brain volume change differentiates normal aging fromMCIrdquo Neurology vol 72 no 22 pp 1906ndash1913 2009

[2] K Franke and C Gaser ldquoLongitudinal changes in individualBrainAGE in healthy aging mild cognitive impairment andAlzheimerrsquos diseaserdquoGeroPsych vol 25 no 4 pp 235ndash245 2012

[3] MA IkramH A VroomanMWVernooij et al ldquoBrain tissuevolumes in the general elderly populationThe Rotterdam ScanStudyrdquo Neurobiology of Aging vol 29 no 6 pp 882ndash890 2008

[4] A Giorgio and N de Stefano ldquoClinical use of brain volumetryrdquoJournal of Magnetic Resonance Imaging vol 37 no 1 pp 1ndash142013

[5] M W Vannier R L Butterfield D Jordan W A Murphy RG Levitt and M Gado ldquoMultispectral analysis of magneticresonance imagesrdquo Radiology vol 154 no 1 pp 221ndash224 1985

[6] M A Balafar A R Ramli M I Saripan and S MashohorldquoReview of brain MRI image segmentation methodsrdquo ArtificialIntelligence Review vol 33 no 3 pp 261ndash274 2010

[7] J C Bezdek L O Hall and L P Clarke ldquoReview of MR imagesegmentation techniques using pattern recognitionrdquo MedicalPhysics vol 20 no 4 pp 1033ndash1048 1993

[8] L P Clarke R P Velthuizen M A Camacho et al ldquoMRIsegmentation methods and applicationsrdquo Magnetic ResonanceImaging vol 13 no 3 pp 343ndash368 1995

[9] J S Duncan X Papademetris J Yang M Jackowski X Zengand L H Staib ldquoGeometric strategies for neuroanatomic ana-lysis from MRIrdquo NeuroImage vol 23 supplement 1 pp S34ndashS45 2004

[10] D L PhamC Xu and J L Prince ldquoCurrentmethods inmedicalimage segmentationrdquoAnnual Review of Biomedical Engineeringvol 2 no 2000 pp 315ndash337 2000

[11] N Saeed ldquoMagnetic resonance image segmentation usingpattern recognition and applied to image registration andquantitationrdquo NMR in Biomedicine vol 11 no 4-5 pp 157ndash1671998

[12] J S Suri S Singh and L Reden ldquoComputer vision andpattern recognition techniques for 2-D and 3-D MR cerebralcortical segmentation (Part I) a state-of-the-art reviewrdquo PatternAnalysis amp Applications vol 5 no 1 pp 46ndash76 2002

[13] A P Zijdenbos and B M Dawant ldquoBrain segmentation andwhite matter lesion detection in MR imagesrdquo Critical Reviewsin Biomedical Engineering vol 22 no 5-6 pp 401ndash465 1994

[14] K Price ldquoAnything you can do I can do better (No you canrsquot)rdquoComputer Vision Graphics and Image Processing vol 36 no 2-3pp 387ndash391 1986

[15] R de Boer H A Vrooman M A Ikram et al ldquoAccuracy andreproducibility study of automatic MRI brain tissue segmenta-tion methodsrdquo NeuroImage vol 51 no 3 pp 1047ndash1056 2010

[16] J de Bresser M P Portegies A Leemans G J Biessels L JKappelle andM A Viergever ldquoA comparison ofMR based seg-mentation methods for measuring brain atrophy progressionrdquoNeuroImage vol 54 no 2 pp 760ndash768 2011

Computational Intelligence and Neuroscience 15

[17] M B Cuadra L Cammoun T Butz O Cuisenaire and J-P Thiran ldquoComparison and validation of tissue modelizationand statistical classification methods in T1-weighted MR brainimagesrdquo IEEE Transactions on Medical Imaging vol 24 no 12pp 1548ndash1565 2005

[18] F Klauschen A Goldman V Barra A Meyer-Lindenbergand A Lundervold ldquoEvaluation of automated brain MR imagesegmentation and volumetrymethodsrdquoHuman BrainMappingvol 30 no 4 pp 1310ndash1327 2009

[19] H Zaidi T Ruest F Schoenahl and M-L Montandon ldquoCom-parative assessment of statistical brain MR image segmentationalgorithms and their impact on partial volume correction inPETrdquo NeuroImage vol 32 no 4 pp 1591ndash1607 2006

[20] J A Frazier S M Hodge J L Breeze et al ldquoDiagnostic and sexeffects on limbic volumes in early-onset bipolar disorder andschizophreniardquo Schizophrenia Bulletin vol 34 no 1 pp 37ndash462008

[21] A Klein and J Tourville ldquo101 labeled brain images and aconsistent human cortical labeling protocolrdquo Frontiers in Neu-roscience vol 6 article 171 2012

[22] B Van Ginneken T Heimann and M Styner ldquo3D segmen-tation in the clinic a grand challengerdquo in Proceedings of the10th International Conference on Medical Image Computingand Computer Assisted Intervention (MICCAI rsquo07) pp 7ndash15Brisbane Australia October-November 2007

[23] A M Dale B Fischl and M I Sereno ldquoCortical surface-based analysis I Segmentation and surface reconstructionrdquoNeuroImage vol 9 no 2 pp 179ndash194 1999

[24] B Fischl M I Sereno and A M Dale ldquoCortical surface-basedanalysis II inflation flattening and a surface-based coordinatesystemrdquo NeuroImage vol 9 no 2 pp 195ndash207 1999

[25] M Jenkinson C F Beckmann T E J BehrensMWWoolrichand S M Smith ldquoFSLrdquoNeuroImage vol 62 no 2 pp 782ndash7902012

[26] J Ashburner and K J Friston ldquoUnified segmentationrdquo Neu-roImage vol 26 no 3 pp 839ndash851 2005

[27] Y D Reijmer A Leemans M Brundel L J Kappelle and GJ Biessels ldquoDisruption of the cerebral white matter network isrelated to slowing of information processing speed in patientswith type 2 diabetesrdquoDiabetes vol 62 no 6 pp 2112ndash2115 2013

[28] S Klein M Staring K Murphy M A Viergever and J P WPluim ldquoElastix a toolbox for intensity-based medical imageregistrationrdquo IEEE Transactions onMedical Imaging vol 29 no1 pp 196ndash205 2010

[29] K J Friston J T Ashburner S J Kiebel T E Nichols and WD Penny Eds Statistical Parametric Mapping The Analysis ofFunctional Brain Images Academic Press 2011

[30] F Ritter T Boskamp A Homeyer et al ldquoMedical imageanalysis a visual approachrdquo IEEE Pulse vol 2 no 6 pp 60ndash702011

[31] L R Dice ldquoMeasures of the amount of ecologic associationbetween speciesrdquo Ecology vol 26 no 3 pp 297ndash302 1945

[32] D P Huttenlocher G A Klanderman and W J RucklidgeldquoComparing images using the Hausdorff distancerdquo IEEE Trans-actions on Pattern Analysis and Machine Intelligence vol 15 no9 pp 850ndash863 1993

[33] A Alansary A Soliman F Khalifa et al ldquoMAP-based frame-work for segmentation of MR brain images based on visualappearance and prior shaperdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal September 2013

[34] A Jog S Roy J L Prince and A Carass ldquoMR brain segmenta-tion using decision treesrdquo in Proceedings of the MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal Nagoya Japan September 2013

[35] R Katyal S Paneri and M Kuse ldquoGaussian intensity modelwith neighborhood cues for fluid-tissue categorization ofmulti-sequence MR brain imagesrdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal Nagoya Japan September 2013

[36] QMahmoodMAlipoor A Chodorowski AMehnert andMPersson ldquoMultimodal MR brain segmentation using Bayesian-based adaptive mean-shift (BAMS)rdquo in Proceedings of theMICCAI WorkshopsmdashThe MICCAI Grand Challenge on MRBrain Image Segmentation (MRBrainS rsquo13) The Midas JournalSeptember 2013

[37] A van Opbroek F van der Lijn andM de Bruijne ldquoAutomatedbrain-tissue segmentation by multi-feature SVM classificationrdquoin Proceedings of the MICCAI WorkshopsmdashThe MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal September 2013

[38] S Pereira J Festa J A Mariz N Sousa and C A SilvaldquoAutomatic brain tissue segmentation of multi-sequence MRimages using random decision forestsrdquo in Proceedings of theMICCAI Grand Challenge on MR Brain Image Segmentation(MRBrainS rsquo13) The Midas Journal Nagoya Japan September2013

[39] MRajchl J Baxter J Yuan T Peters andAKhan ldquoMulti-atlas-based segmentation with hierarchical max-flowrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[40] D Sarikaya L Zhao and J J Corso ldquoMulti-atlas brain MRIsegmentation with multiway cutrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[41] H Vrooman F van der Lijn and W Niessen ldquoAuto-kNNbrain tissue segmentation using automatically trained k-nearest-neighbor classificationrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[42] S Vyas P Burlina D Kleissas and R Mukherjee ldquoAutomatedwalks using machine learning for segmentationrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[43] C Wang and O Smedby ldquoFully automatic brain segmentationusing model-guided level sets and skeleton-based modelsrdquo inProceedings of theMICCAI Grand Challenge onMRBrain ImageSegmentation (MRBrainS rsquo13) The Midas Journal NagoyaJapan September 2013

[44] C Cortes and V Vapnik ldquoSupport-vector networksrdquo MachineLearning vol 20 no 3 pp 273ndash297 1995

[45] H A Vrooman C A Cocosco F van der Lijn et al ldquoMulti-spectral brain tissue segmentation using automatically trainedk-Nearest-Neighbor classificationrdquo NeuroImage vol 37 no 1pp 71ndash81 2007

[46] C Wang H Frimmel and O Smedby ldquoFast level-set basedimage segmentation using coherent propagationrdquo MedicalPhysics vol 41 no 7 Article ID 073501 2014

16 Computational Intelligence and Neuroscience

[47] A Jog S Roy A Carass and J L Prince ldquoMagnetic resonanceimage synthesis through patch regressionrdquo in Proceedings of theIEEE 10th International Symposium onBiomedical Imaging (ISBIrsquo13) pp 350ndash353 April 2013

[48] J G Sied A P Zijdenbos and A C Evans ldquoA non-parametricmethod for automatic correction of intensity non-uniformity inMRI datardquo IEEE Transactions on Medical Imaging vol 17 no 1pp 87ndash97 1998

[49] A Carass J Cuzzocreo M B Wheeler P-L Bazin S MResnick and J L Prince ldquoSimple paradigm for extra-cerebraltissue removal algorithm and analysisrdquoNeuroImage vol 56 no4 pp 1982ndash1992 2011

[50] M Modat G R Ridgway Z A Taylor et al ldquoFast free-form deformation using graphics processing unitsrdquo ComputerMethods and Programs in Biomedicine vol 98 no 3 pp 278ndash284 2010

[51] M Rajchl J Yuan J A White et al ldquoInteractive hierarchical-flow segmentation of scar tissue from late-enhancement cardiacmr imagesrdquo IEEE Transactions on Medical Imaging vol 33 no1 pp 159ndash172 2014

[52] QMahmood A Chodorowski AMehnert andM Persson ldquoAnovel Bayesian approach to adaptive mean shift segmentationof brain imagesrdquo in Proceedings of the 25th IEEE InternationalSymposiumonComputer-BasedMedical Systems (CBMS rsquo12) pp1ndash6 June 2012

[53] A El-Baz M Nitzken A Alansary A Soliman and MCasanova ldquoBrain Segmentation Method for Young Childrenand Adultsrdquo US Provisional Patent Application 13100

[54] A Alansary A Soliman M Nitzken et al ldquoAn integratedgeometrical and stochastic approach for accurate infant brainextractionrdquo in Proceedings of the IEEE International Conferenceon Image Processing (ICIP rsquo14) pp 3542ndash3546 October 2014

[55] V Popescu M Battaglini W S Hoogstrate et al ldquoOptimizingparameter choice for FSL-Brain Extraction Tool (BET) on 3DT1 images in multiple sclerosisrdquo NeuroImage vol 61 no 4 pp1484ndash1494 2012

[56] J L Whitwell W R Crum H C Watt and N C FoxldquoNormalization of cerebral volumes by use of intracranial vol-ume implications for longitudinal quantitative MR imagingrdquoAmerican Journal of Neuroradiology vol 22 no 8 pp 1483ndash1489 2001

[57] S M Smith ldquoFast robust automated brain extractionrdquo HumanBrain Mapping vol 17 no 3 pp 143ndash155 2002

[58] D W Shattuck G Prasad M Mirza K L Narr and A WToga ldquoOnline resource for validation of brain segmentationmethodsrdquo NeuroImage vol 45 no 2 pp 431ndash439 2009

[59] I Isgum M J N L Benders B Avants et al ldquoEvaluation ofautomatic neonatal brain segmentation algorithms the Neo-BrainS12 challengerdquo Medical Image Analysis vol 20 no 1 pp135ndash151 2015

[60] P Moeskops M J N L Benders S M Chita et al ldquoAutomaticsegmentation of MR brain images of preterm infants usingsupervised classificationrdquo NeuroImage 2015

[61] P Moeskops M A Viergever M J N L Benders and IIsgum ldquoEvaluation of an automatic brain segmentationmethoddeveloped for neonates on adult MR brain imagesrdquo in MedicalImaging Image Processing vol 9413 of Proceedings of SPIEInternational Society for Optics and Photonics Orlando FlaUSA February 2015

[62] L Wang Y Gao F Shi et al ldquoLINKS learning-based multi-source IntegratioN frameworK for Segmentation of infant brainimagesrdquo NeuroImage vol 108 pp 160ndash172 2015

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

4 Computational Intelligence and Neuroscience

(a) (b) (c) (d) (e)

Figure 1 Example of themanually drawn contours (a) the resulting hard segmentationmap (GM light blueWM yellow andCSF dark blue)that is used as the reference standard (b) the T1-weighted scan (c) the T1-weighted inversion recovery (IR) scan (d) and the T2-weightedfluid attenuated inversion recovery (FLAIR) scan (e)

For each component 119888 isin 119862 and each evaluation measure119898 isin119872 the mean and standard deviation are determined over all15 test datasets The segmentation algorithms are then sortedon the mean 119863 value in descending order and on the mean11986795

and AVD value in ascending order Each segmentationalgorithm receives a rank (119903) between 1 (ranked best) and 119899(number of participating algorithms) for each component 119888and each evaluation measure 119898 The final ranking is basedon the overall score of each algorithm which is the sum overall ranks defined as

119904 =

|119872|

sum

119898=0

|119862|

sum

119888=0

119903119898119888 (5)

where 119903119898119888

is the rank of the segmentation algorithm formeasure 119898 of component 119888 For the final ranking 119903 theoverall scores 119904 are sorted in ascending order and rankedfrom 1 to 119899 In case two or more algorithms have equalscores the standard deviation over all 15 test datasets is takeninto account to determine the final rank The segmentationalgorithms are then sorted on the standard deviation inascending order and ranked for each component 119888 and eachevaluation measure119898 The overall score is determined using(5) and the algorithms are sorted based on this score inascending order and ranked from 1 to 119899 The algorithms thathave equal overall scores based on the mean value are thenranked based on this standard deviation rank

22 EvaluatedMethods Theevaluation framework describedin Section 21 was launched at the MRBrainS13 challengeworkshop at the Medical Image Computing and ComputerAssisted Intervention (MICCAI) conference on September26th in 2013 For the workshop challenge the test datasetswere split into twelve off-site and three on-site test datasetsFor the off-site part teams could register on the MRBrainSwebsite (httpmrbrains13isiuunl) and download the fivetraining and twelve test datasets A time slot of eight weekswas available for teams to download the data train theiralgorithms segment the test datasets and submit their resultson the website Fifty-eight teams downloaded the data ofwhich twelve submitted their segmentation results The eval-uation results were reported to the twelve teams and all teamssubmitted a workshop paper to the MRBrainS13 challengeworkshop at MICCAI Eleven teams presented their results

at the workshop and segmented the three on-site test datasetslive at the workshop within a time slot of 35 hours Thesealgorithms provide a benchmark for the proposed evaluationframework and are briefly described in Sections 221ndash2211in alphabetical order of teamsrsquo names The teamsrsquo namesare used in the paper to identify the methods For a fulldescription of the methods we refer to the workshop papers[33ndash43] In Section 2212 we describe the evaluated freewarepackages

221 BIGR2 [37] This multifeature SVM [37] method clas-sifies voxels by using a Support Vector Machine (SVM)classifier [44] with a Gaussian kernel Besides spatial featuresand intensity information from all three MRI sequences theSVM classifier incorporates Gaussian-scale-space features tofacilitate a smooth segmentation Skull stripping is performedby nonrigid registration of the masks of the training imagesto the target image

222 Bigr neuro [41] This auto-kNN [45] method is basedon an automatically trained kNN-classifier First a proba-bilistic tissue atlas is generated by nonrigidly registering themanually annotated atlases to the subject of interest Trainingsamples are obtained by thresholding the probabilistic atlasand subsequently pruning the feature space White matterlesions are detected by applying an adaptive threshold deter-mined from the tissue segmentation to the FLAIR sequence

223 CMIV [43] A statistical-model-guided level-setmethod is used to segment the skull brain ventricles andbasal ganglia Then a skeleton-based model is created byextracting the midsurface of the gray matter and definingthe thickness This model is incorporated into a level-setframework to guide the cortical gray matter segmentationThe coherent propagation algorithm [46] is used to acceleratethe level-set evolution

224 Jedi MindMeld [42] Thismethod starts by preprocess-ing the data via anisotropic diffusion For each 2D slice of alabeled dataset the canny edge pixels are extracted and theTourist Walk is computed This is done for axial sagittal andcoronal views Machine learning is used with these featuresto automatically label edge pixels in an unlabeled dataset

Computational Intelligence and Neuroscience 5

Finally these labels are used by the Random Walker forautomatic segmentation

225 LNMBrains [35] The voxel intensities of all MRIsequences are modelled as a Gaussian distribution for eachlabel The parameters of the Gaussian distributions areevaluated asmaximum likelihood estimates and the posteriorprobability of each label is determined by using Bayesianestimation A feature set consisting of regional intensitytexture spatial location of voxels and the posterior prob-ability estimates is used to classify each voxel into CSFWM GM or background by using a multicategory SVMclassifier

226 MNAB [38] This method uses Random DecisionForests to classify the voxels into GM WM and CSF Itstarts by a skull stripping procedure followed by an intensitynormalization of each MRI sequence Feature extraction isthen performed on the intensities posterior probabilitiesneighborhood statistics tissue atlases and gradient mag-nitude After classification isolated voxels are removed bypostprocessing

227 Narsil [34] This is a model-free algorithm that usesensembles of decision trees [47] to learn the mapping fromimage features to the corresponding tissue label The ensem-bles of decision trees are constructed from correspondingimage patches of the provided T1 and FLAIR scans withmanual segmentations The N3 algorithm [48] was used foradditional inhomogeneity correction and SPECTRE [49] wasused for skull stripping

228 Robarts [39] Multiatlas registration [50] with the T1training images was used to propagate labels to generatesample histograms in a log-likelihood intensity model andprobabilistic shape priors These were employed in a MAPdata term and regularized via computation of a hierar-chical max-flow [51] A brain mask from registration ofthe T1-IR training images was used to obtain the finalresults

229 S2 QM [36] This method [52] is based on Bayesian-based adaptive mean shift and the voxel-weighted 119870-meansalgorithm The former is used to segment the brain into alarge number of clusters or modes The latter is employed toassign these clusters to one of the three components WMGM or CSF

2210 UB VPMLMed [40] This method creates a multiatlasby registering the training images to the subject imageand then propagating the corresponding labels to a fullyconnected graph on the subject image Label fusion thencombines the multiple labels into one label at each voxelwith intensity similarity based weighted voting Finally themethod clusters the graph using multiway cut in order toachieve the final segmentation

2211 UofL BioImaging [33] This is an automated MAP-based method aimed at unsupervised segmentation of differ-ent brain tissues from T1-weighted MRI It is based on theintegration of a probabilistic shape prior a first-order inten-sity model using a Linear Combination of Discrete Gaussians(LCDG) and a second-order appearance model These threefeatures are integrated into a two-level joint Markov-GibbsRandom Field (MGRF) model of T1-MR brain imagesSkull stripping was performed using BET2 [40] followed byan adaptive threshold-based technique to restore the outerborder of the CSF using both T1 andT1-IR this techniquewasnot described in [33] due to a US patent application [53] butis described in [54] This method was applied semiautomat-ically to the MRBrainS test data due to per scan parametertuning

2212 Freeware Packages Next to the methods evaluatedat the workshop we evaluated three commonly used free-ware packages for MR brain image segmentation FreeSurfer(httpsurfernmrmghharvardedu) [23 24] FSL (httpfslfmriboxacukfslfslwiki) [25] and SPM12 (httpwwwfilionuclacukspm) [26] All packages were applied usingthe default settings unless mentioned otherwise FreeSurfer(v530) was applied to the high resolution T1 sequence Themri label2vol tool was used to map the labels on the thickslice T1 that was used for the evaluation FSL (v50) wasdirectly applied to the thick slice T1 and provides both apveseg and a seg file as binary output We evaluated bothof these files The fractional intensity threshold parameterldquo119891rdquo of the BET tool that sets the brainnonbrain intensitythreshold was set according to [55] at 02 (Philips Achieva3T setting) SPM12 was directly applied to the thick sliceT1 sequence as well However it also provides the optionto add multiple MRI sequences Therefore we evaluatedSPM12 not only on the thick slice T1 sequence but addedthe T1-IR and the T2-FLAIR scan as well and tested variouscombinationsThe number of Gaussians was set according tothe SPM manual to two for GM two for WM and two forCSF

23 Statistical Analysis All evaluated methods werecompared to the reference standard In summary ofthe results the mean and standard deviation overall 15 test datasets were calculated per component(GM WM and CSF) and combination of components(brain intracranial volume) and per evaluation measure(Dice 95th-percentile Hausdorff distance and absolutevolume difference) for each of the evaluated methodsBoxplots were created using R version 303 (R projectfor statistical computing (httpwwwr-projectorg))Since white matter lesions should be segmented as whitematter the percentage of white matter lesion voxelssegmented as white matter (sensitivity) was calculatedfor each algorithm over all 15 test datasets to evaluatethe robustness of the segmentation algorithms againstpathology

6 Computational Intelligence and Neuroscience

3 Results

Table 1 presents the final ranking (119903) of the evaluatedmethodsthat participated in the workshop as well as the evalu-ated freeware packages During the workshop team UofLBioImaging ranked first and BIGR2 ranked second with onepoint difference in the overall score 119904 (5) However addingthe results of the freeware packages resulted in an equal scorefor UofL BioImaging and BIGR2 Therefore the standarddeviation rank was taken into account and BIGR2 is rankedfirst with standard deviation rank four and UofL BioImagingis ranked second with standard deviation rank eight Table 1further presents the mean standard deviation and rank foreach evaluation measure (119863119867

95 and AVD) and component

(GM WM and CSF) as well as the brain (WM + GM) andintracranial volume (WM+GM+CSF) Team BIGR2 scoredbest for theGMWM and brain segmentation and teamUofLBioImaging for the CSF segmentation Team Robarts scoredbest for the intracranial volume segmentation The boxplotsfor all evaluation measures and components are shown inFigures 2ndash4 and include the results of the freeware packagesFigure 5 shows an example of the segmentation results atthe height of the basal ganglia (slice 22 of test subject 9)The sensitivity of the algorithms to segment white matterlesions as WM and examples of the segmentation results inthe presence of white matter lesions (slice 31 of test subject3) are shown in Figure 6 Team UB VPML Med scores thehighest sensitivity of white matter lesions segmented as whitematter and is therefore most robust in the presence of thistype of pathology

4 Discussion

In this paper we proposed the MRBrainS challenge onlineevaluation framework to evaluate automatic and semiauto-matic algorithms for segmenting GM WM and CSF on3T multisequence (T1 T1-IR and T2-FLAIR) MRI scans ofthe brain We have evaluated and presented the results ofeleven segmentation algorithms that provide a benchmarkfor algorithms that will use the online evaluation frameworkto evaluate their performance Team UofL BioImaging andBIGR2 have equal overall scores but BIGR2 was rankedfirst based on the standard deviation ranking The evalu-ated methods represent a wide variety of algorithms thatinclude Markov random field models clustering approachesdeformable models and atlas-based approaches and classi-fiers (SVM KNN and decision trees) The presented evalua-tion framework provides an insight into the performance ofthese algorithms in terms of accuracy and robustness Variousfactors influence the choice for a certain method aboveothers We provide three measures that could aid in selectingthe method that is most appropriate for the segmentationgoal at hand a boundarymeasure (95th-percentile Hausdorffdistance 119867

95) an overlap measure (Dice coefficient 119863) and

a volume measure (absolute volume difference AVD) Allthree measures are taken into account for the final rankingof the methods This ranking was designed to get a quickinsight into how the methods perform in comparison to eachother The best overall method is the method that performs

well for all three measures and all three components (GMWM andCSF) However whichmethod to select depends onthe segmentation goal at hand Not all measures are relevantfor all segmentation goals For example if segmentation isused for brain volumetry [4] the overlap (119863) and volume(AVD) measures of the brain and intracranial volume (usedfor normalization [56]) segmentations are important to takeinto account On the other hand if segmentation is usedfor cortical thickness measurements the focus should be onthe gray matter boundary (119867

95) and overlap (119863) measures

Therefore the final ranking should be used to get a first insightinto the overall performance after which the performanceof the measures and components that are most relevant forthe segmentation goal at hand should be considered Besidesaccuracy robustness could also influence the choice for acertain method above others For example team UB VPMLMed shows a high sensitivity score for segmenting whitemat-ter lesions as white matter (Figure 6) and shows a consistentsegmentation performance of gray and white matter over all15 test datasets (Figures 2ndash4) This could be beneficial forsegmenting scans of populations with white matter lesionsbut is less important if the goal is to segment scans ofyoung healthy subjects In the latter case the most accuratesegmentation for gray and white matter (team BIGR2) ismore interesting If a segmentation algorithm is to be usedin clinical practice speed is an important consideration aswell The runtime of the evaluated methods is reported inTable 1 However these runtimes are merely an indicationof the required time since academic software is generallynot optimized for speed and the runtime is measured ondifferent computers andplatformsAnother relevant aspect ofthe evaluation framework is the comparison of multi- versussingle-sequence approaches For example most methodsstruggle with the segmentation of the intracranial volume onthe T1-weighted scan There is no contrast between the CSFand the skull and the contrast between the dura mater andthe CSF is not always sufficient Team Robarts used an atlas-based registration approach on the T1-IR scan (good contrastbetween skull and CSF) to segment the intracranial volumewhich resulted in the best performance for intracranialvolume segmentation (Table 1 Figures 2ndash4) Most methodsadd the T2-FLAIR scan to improve robustness against whitematter lesions (Table 1 Figure 6) Although using only theT1-weighted scan and incorporating prior shape information(team UofL BioImaging) can be very effective also thefreeware packages support this as well Since FreeSurfer isan atlas-based method it uses prior information and is themost robust of all freeware packages to white matter lesionsHowever adding the T2 FLAIR scan to SPM12 increasesrobustness against white matter lesions as well as comparedto applying SPM12 to the T1 scan only (Figure 6) In generalSPM12with the T1 and the T2-FLAIR sequence performswellin comparison to the other freeware packages (Table 1 andFigures 2ndash4) on the thick slice MRI scans Although addingthe T1-IR scan to SPM increases the performance of the CSFand ICV segmentations as compared to using only the T1 andT2-FLAIR sequence it decreases the performance of the GMand WM segmentations Therefore adding all sequences toSPM12 did not result in a better overall performance

Computational Intelligence and Neuroscience 7

Table 1 Results of the 11 evaluated algorithms presented at the workshop and the evaluated freeware packages on the 15 test datasets Thealgorithms are ranked (119903) based on their overall score (119904) by using (5) This score is based on the ranks of the gray matter (GM) white matter(WM) and cerebrospinal fluid (CSF) segmentation and the three evaluated measures Dice coefficient (119863 in ) 95th-percentile Hausdorffdistance (119867

95in mm) and the absolute volume difference (AVD in ) The rank 119903

119898119888denotes the rank based on the mean (120583) over all 15 test

datasets for each measure 119898 (0 119863 1 11986795 and 2 AVD) and component 119888 (0 GM 1 WM 2 CSF 3 brain (WM + GM) and 4 intracranial

volume (ICV = WM + GM + CSF)) Teams BIGR2 and UofL BioImaging and FreeSurfer and Jedi Mind Meld have equal scores based onthe mean (120583) therefore the ranking based on the standard deviation (120590) is taken into account to determine the final rank (BIGR2 120590 rank 4UofL BioImaging 120590 rank 8 FreeSurfer 120590 rank 13 and Jedi MindMeld 120590 rank 17) Columns 2 and 3 present the average runtime 119905 per scan inseconds (s) minutes (m) or hours (h) and the scans (T1 T1-weighted scan 3D T1 3D T1-weighted scan IR T1-weighted inversion recovery(IR) and F T1-weighted FLAIR) that are used for processing

119903 Team 119905 Scans

GM WM CSF

119904

Brain ICV119903001199031011990320119903011199031111990321119903021199031211990322

119903031199031311990323119903041199031411990324

119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590)

1 BIGR2 35m T1 IR F1 2 4 2 2 4 4 5 14

381 2 12 8 4 12

847 19 61 884 24 60 783 32 23 951 27 32 960 39 52(13) (04) (33) (12) (05) (51) (50) (08) (17) (05) (08) (16) (13) (11) (30)

2 UofLBioImaginglowast 6 s T1

5 1 9 4 1 13 2 2 138

2 1 12 5 2 5830 17 86 879 22 87 789 27 97 949 24 39 967 34 18(15) (03) (54) (20) (06) (66) (42) (05) (10) (06) (05) (20) (08) (06) (20)

3 CMIV 3m T1 F6 7 5 5 3 10 3 3 8

505 7 2 4 3 11

824 27 68 877 24 73 786 30 14 945 38 26 968 38 49(14) (04) (40) (16) (04) (38) (31) (04) (59) (05) (11) (22) (08) (13) (23)

4 UB VPMLMed 30m T1 IR F

4 4 2 1 5 7 8 13 1761

4 4 1 10 11 15833 21 59 886 27 71 748 43 31 946 28 24 948 66 77(13) (03) (53) (17) (04) (38) (71) (17) (19) (06) (04) (18) (20) (20) (41)

5 Bigr neuro 2 h T1 F7 13 3 6 6 9 6 4 10

647 10 10 7 5 8

815 37 59 873 30 73 782 32 16 940 46 36 963 39 35(17) (09) (42) (14) (04) (38) (47) (06) (14) (08) (14) (24) (12) (09) (27)

6 Robarts 16m 3D T1 IR11 3 15 8 8 6 1 1 13

6616 3 18 1 1 1

797 20 98 862 31 71 803 27 20 931 28 79 979 26 09(24) (01) (73) (13) (04) (62) (41) (05) (13) (16) (05) (36) (03) (04) (07)

7 Narsil 2m T1 F3 5 1 7 11 2 17 18 7

713 5 3 16 18 9

835 23 55 871 33 58 666 133 14 948 29 29 925 24 37(18) (04) (44) (13) (09) (53) (24) (54) (95) (05) (05) (20) (05) (89) (17)

8 SPM T1 F 3m T1 F8 9 16 9 7 1 9 14 2

759 14 14 6 14 4

812 29 10 860 30 52 741 46 10 939 58 53 966 82 15(22) (03) (85) (15) (01) (38) (34) (06) (47) (10) (22) (38) (02) (30) (10)

9 SPM T1 IR 3m T1 IR12 11 7 16 12 5 5 10 3

818 11 7 2 8 2

794 30 72 835 36 63 783 40 10 939 46 34 977 65 10(21) (04) (63) (21) (03) (46) (38) (06) (57) (08) (12) (28) (02) (13) (08)

10 MNAB 15m T1 IR F2 8 12 3 4 11 15 15 16

866 9 11 15 12 17

839 28 91 880 27 78 681 49 29 945 45 38 925 71 97(21) (09) (65) (12) (08) (40) (40) (22) (21) (10) (20) (32) (11) (42) (47)

11 SPM T1 3m T19 10 6 11 10 3 11 16 15

9110 8 6 9 13 13

803 30 69 856 31 60 707 53 23 939 44 32 953 81 55(24) (05) (68) (17) (01) (41) (38) (15) (157) (09) (16) (29) (09) (37) (37)

12 FSL Seg 10m T113 16 10 10 13 14 12 6 5

9913 13 4 12 6 6

787 43 86 860 37 115 699 34 12 933 55 30 942 53 34(22) (12) (63) (26) (08) (63) (28) (02) (103) (08) (14) (15) (08) (11) (15)

8 Computational Intelligence and Neuroscience

Table 1 Continued

119903 Team 119905 Scans

GM WM CSF

119904

Brain ICV119903001199031011990320119903011199031111990321119903021199031211990322

119903031199031311990323119903041199031411990324

119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590)

13 SPM T1 IR F 4m T1 IR F10 12 18 15 14 12 7 11 6

10512 15 13 3 15 3

801 30 139 836 38 84 769 41 12 936 59 51 977 82 12(24) (02) (96) (21) (05) (52) (31) (05) (60) (11) (18) (36) (02) (18) (09)

14 FSL PVSeg 10m T115 15 8 13 15 17 13 7 4

10711 16 9 13 7 7

777 43 84 848 38 197 695 34 11 936 61 35 942 53 34(26) (13) (65) (32) (09) (10) (22) (03) (58) (09) (16) (30) (08) (11) (15)

15 FreeSurfer 1 h 3D T116 6 17 12 9 8 18 12 18

11617 6 16 17 10 18

774 23 121 852 31 72 658 43 50 923 36 63 924 66 10(20) (06) (60) (22) (05) (46) (37) (06) (196) (08) (06) (34) (09) (04) (45)

16 Jedi MindMeld 27 s T1 IR F

14 14 11 17 16 15 10 8 11116

15 12 8 11 16 14778 39 89 806 45 137 733 37 18 932 54 35 943 20 68(59) (28) (76) (96) (36) (12) (35) (08) (90) (17) (48) (30) (17) (37) (31)

17 S2 QM 15 h T1 IR F17 17 13 14 17 18 16 9 9

13014 17 15 14 9 10

764 55 93 839 49 248 679 38 14 933 70 55 937 65 40(34) (30) (64) (34) (32) (10) (23) (06) (99) (14) (35) (45) (11) (14) (22)

18 LNMBrains 5m T1 IR F18 18 14 18 18 16 14 17 12

14518 18 17 18 17 16

728 68 95 783 68 153 688 77 20 885 86 74 892 204 79(53) (23) (76) (64) (35) (13) (66) (24) (13) (45) (28) (77) (48) (37) (83)

lowastSemiautomatic due to per scan parameter tuning

Besides the advantages of the MRBrainS evaluationframework there are some limitations that should be takeninto account The T1-weighted IR and the T2-weightedFLAIR scan were acquired with a lower resolution (096times 096 times 300mm3) than the 3D T1-weighted scan (10 times10 times 10mm3) To be able to provide a registered multise-quence dataset the 3D T1-weighted scan was registered tothe T2-weighted FLAIR scan and downsampled to 096 times096 times 300mm3 The reference standard is therefore onlyavailable for this resolution The decreased performance ofthe FreeSurfer GM segmentation as compared to the otherfreeware packages might be due to the fact that we evaluateon the thick slice T1 sequence instead of the high resolutionT1 Performing the manual segmentations to provide thereference standard is very laborsome and time consumingInstead of letting multiple observers manually segment theMRI datasets or letting one observer manually segmentthe MRI datasets twice much time and effort was spenton creating one reference standard that was as accurate aspossible Therefore we were not able to determine the inter-or intraobserver variability Finally we acknowledge that ourevaluation framework is limited to evaluating the accuracyand robustness over 15 datasets for segmenting GM WMand CSF on 3T MRI scans acquired on a Philips scanner ofa specific group of elderly subjects Many factors influencesegmentation algorithm performance such as the type ofscanner (vendor field strength) the acquisition protocolthe available MRI sequences and the type of subjects

Participating algorithmsmight have been designed for differ-ent types of MRI scans Therefore the five provided trainingdatasets are important for participants to be able to traintheir algorithms on the provided data Some algorithms aredesigned to segment only some components such as onlyGM and WM instead of all three components and usefreely available software such as the brain extraction tool[57] to segment the outer border of the CSF (intracranialvolume) We have chosen to base the final ranking on allthree components but it is therefore important to assess notonly the final ranking but the performance of the individualcomponents as well

Despite these limitations the MRBrainS evaluationframework provides an objective and direct comparison ofsegmentation algorithms The reference standard of the testdata is unknown to the participants the same evaluationmeasures are used for all evaluated algorithms and partici-pants apply their own algorithms to the provided data

In comparison to the online validation engine proposedby Shattuck et al [58] the MRBrainS evaluation frameworkuses 3T MRI data instead of 15T MRI data and evaluates notonly brain versus nonbrain segmentation but also segmenta-tion of gray matter white matter cerebrospinal fluid brainand intracranial volume The availability of many differenttypes of evaluation frameworks will aid in the developmentof more generic and robust algorithms For example inthe NEATBrainS (httpneatbrains15isiuunl) challengeresearchers were challenged to apply their algorithms to data

Computational Intelligence and Neuroscience 9

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

65

70

75

80

85

GM Dice

Dic

e (

)

(a)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

55

60

65

70

75

80

85

90

WM Dice

Dic

e (

)

(b)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

55

60

65

70

75

80

85CSF Dice

Dic

e (

)

(c)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

80

85

90

95

Brain (WM + GM) Dice

Dic

e (

)

(d)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

80

85

90

95

Dic

e (

)

Intracranial volume (WM + GM + CSF) Dice

(e)

Figure 2 Boxplots presenting the evaluation results for the Dice coefficient (1) of the gray matter (GM) white matter (WM) cerebrospinalfluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all 15 test datasets

10 Computational Intelligence and Neuroscience

2468

10121416

GM H-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(a)

2468

10121416

WMH-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(b)

5

10

15

20CSF H-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(c)

5

10

15

20H

-95

(mm

)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Brain (WM + GM) H-95

(d)

5

10

15

20

25

30

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Intracranial volume (WM + GM + CSF) H-95

(e)

Figure 3 Boxplots presenting the evaluation results for the 95th-percentile Hausdorff distance (3) of the gray matter (GM) white matter(WM) cerebrospinal fluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all15 test datasets

Computational Intelligence and Neuroscience 11

0

10

20

30

GM AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(a)

0

10

20

30

40WM AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(b)

0

20

40

60

80

CSF AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(c)

0

5

10

15

20

25

30AV

D (

)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Brain (WM + GM) AVD

(d)

0

5

10

15

20

25

30

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Intracranial volume (WM + GM + CSF) AVD

(e)

Figure 4 Boxplots presenting the evaluation results for the absolute volume difference (4) of the gray matter (GM) white matter (WM)cerebrospinal fluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all 15 testdatasets

12 Computational Intelligence and Neuroscience

T2

UofL

N

FS

S2

‐FLAIR

BioImaging

Narsil

L‐Seg

2_QM

T1‐IR

CMIV

SPM12_T1

SPM12_T1_

LNMBr

_FLAIR

_IR_FLAIR

ains

T1

UB VPML Me

SPM12_T1_ I

FSL‐PVSeg

R

d B

R

F

eference

igr_neuro

MNAB

reeSurfer

BIG

Rob

SPM1

Jedi Mi

R2

arts

2_T1

nd Meld

Figure 5 Illustration of the segmentation results at the height of the basal ganglia (test subject 9 slice 22) The basal ganglia should besegmented as gray matter The first three images show the three MRI sequences The fourth image (reference) is the manually segmentedreference standard (yellow white matter light blue gray matter and dark blue cerebrospinal fluid) The results of the participatingsegmentation algorithms are ordered from the best overall performance (BIGR2) to the worst overall performance (LNMBrains) Majordifferences are present in the segmentation of the basal ganglia and the outer border of the cerebrospinal fluid The arrows indicate examplelocations where the segmentation results differ from the ground truth

Computational Intelligence and Neuroscience 13

T2

UofL BioIm

Nars

FSL‐S

S2_Q

‐FLAIR

aging (860)

il (892)

eg (453)

M (565)

T1‐IR

CMIV (90

SPM12_T1_F

SPM12_T1_IR

LNMBrains

5) UB

(831) SP

_F (815) F

(286)

T1

VPML Med (9

M12_T1_IR (60

SL‐PVSeg (525

R

10) Bigr_

5) MN

) FreeS

eference

neuro (896)

AB (889)

urfer (846)

BIGR2

Robarts

SPM12_T

Jedi Mind M

(852)

(810)

1 (683)

eld (769)

Figure 6 Illustration of the segmentation results in the presence of whitematter lesions (test subject 3 slice 31)Whitematter lesions (WMLs)should be labeled asWM the sensitivity (percentage ofWML voxels is labeled asWM) over all 15 datasets is presented between brackets afterthe team nameThe first three images show the threeMRI sequences T2-weighted fluid attenuated inversion recovery T1-weighted inversionrecovery and T1-weighted scanThe fourth image (reference) is themanually segmented reference standard (red whitematter lesions yellowwhite matter light blue gray matter and dark blue cerebrospinal fluid)The results of the segmentation algorithms are ordered from the bestoverall performance (BIGR2) to the worst overall performance (LNMBrains)The arrows indicate example locations where the segmentationresults differ from the ground truth

14 Computational Intelligence and Neuroscience

fromboth theMRBrainS and theNeoBrainS [59] (brain tissuesegmentation in neonates) challenge Two methods [60ndash62]specifically designed for neonatal brain tissue segmentationshowed a high performance for tissue segmentation on theMRBrainS data Applying algorithms to different types of datahas the potential to lead to new insights and more robustalgorithms The MRBrainS evaluation framework remainsopen for new contributions At the time of writing 21teams had submitted their results on the MRBrainS website(httpmrbrains13isiuunlresultsphp)

5 Conclusion

The MRBrainS challenge online evaluation framework pro-vides direct and objective comparison of automatic andsemiautomatic methods to segment GM WM CSF brainand ICV on 3T multisequence MRI data The first elevenparticipating methods are evaluated and presented in thispaper as well as three commonly used freeware packages(FreeSurfer FSL and SPM12) They provide a benchmarkfor future contributions to the framework The final rankingprovides a quick insight into the overall performance of theevaluated methods in comparison to each other whereasthe individual evaluation measures (Dice 95th-percentileHausdorff distance and absolute volume difference) percomponent (GM WM CSF brain and ICV) can aid inselecting the best method for a specific segmentation goal

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was financially supported by IMDI Grant104002002 (Brainbox) fromZonMw the Netherlands Organ-isation for Health Research and Development within kindsponsoring by Philips theUniversityMedical CenterUtrechtand Eindhoven University of TechnologyThe authors wouldlike to acknowledge the following members of the UtrechtVascular Cognitive Impairment Study Group who were notincluded as coauthors of this paper but were involved in therecruitment of study participants and MRI acquisition at theUMC Utrecht (in alphabetical order by department) E vanden Berg M Brundel S Heringa and L J Kappelle of theDepartment of Neurology P R Luijten and W P Th MMali of the Department of Radiology and A Algra and GE H M Rutten of the Julius Center for Health Sciences andPrimary CareThe research of Geert Jan Biessels and the VCIgroupwas financially supported byVIDIGrant 91711384 fromZonMw and by Grant 2010T073 of the Netherlands HeartFoundation The research of Jeroen de Bresser is financiallysupported by a research talent fellowship of the Univer-sity Medical Center Utrecht (Netherlands) The research ofAnnegreet vanOpbroek andMarleen de Bruijne is financiallysupported by a research grant from NWO (the NetherlandsOrganisation for Scientific Research) The authors would

like to acknowledge MeVis Medical Solutions AG (BremenGermany) for providing MeVisLab Duygu Sarikaya andLiang Zhao acknowledge their Advisor Professor JasonCorsofor his guidance Duygu Sarikaya is supported by NIH 1 R21CA160825-01 and Liang Zhao is partially supported by theChina Scholarship Council (CSC)

References

[1] I Driscoll C Davatzikos Y An et al ldquoLongitudinal pattern ofregional brain volume change differentiates normal aging fromMCIrdquo Neurology vol 72 no 22 pp 1906ndash1913 2009

[2] K Franke and C Gaser ldquoLongitudinal changes in individualBrainAGE in healthy aging mild cognitive impairment andAlzheimerrsquos diseaserdquoGeroPsych vol 25 no 4 pp 235ndash245 2012

[3] MA IkramH A VroomanMWVernooij et al ldquoBrain tissuevolumes in the general elderly populationThe Rotterdam ScanStudyrdquo Neurobiology of Aging vol 29 no 6 pp 882ndash890 2008

[4] A Giorgio and N de Stefano ldquoClinical use of brain volumetryrdquoJournal of Magnetic Resonance Imaging vol 37 no 1 pp 1ndash142013

[5] M W Vannier R L Butterfield D Jordan W A Murphy RG Levitt and M Gado ldquoMultispectral analysis of magneticresonance imagesrdquo Radiology vol 154 no 1 pp 221ndash224 1985

[6] M A Balafar A R Ramli M I Saripan and S MashohorldquoReview of brain MRI image segmentation methodsrdquo ArtificialIntelligence Review vol 33 no 3 pp 261ndash274 2010

[7] J C Bezdek L O Hall and L P Clarke ldquoReview of MR imagesegmentation techniques using pattern recognitionrdquo MedicalPhysics vol 20 no 4 pp 1033ndash1048 1993

[8] L P Clarke R P Velthuizen M A Camacho et al ldquoMRIsegmentation methods and applicationsrdquo Magnetic ResonanceImaging vol 13 no 3 pp 343ndash368 1995

[9] J S Duncan X Papademetris J Yang M Jackowski X Zengand L H Staib ldquoGeometric strategies for neuroanatomic ana-lysis from MRIrdquo NeuroImage vol 23 supplement 1 pp S34ndashS45 2004

[10] D L PhamC Xu and J L Prince ldquoCurrentmethods inmedicalimage segmentationrdquoAnnual Review of Biomedical Engineeringvol 2 no 2000 pp 315ndash337 2000

[11] N Saeed ldquoMagnetic resonance image segmentation usingpattern recognition and applied to image registration andquantitationrdquo NMR in Biomedicine vol 11 no 4-5 pp 157ndash1671998

[12] J S Suri S Singh and L Reden ldquoComputer vision andpattern recognition techniques for 2-D and 3-D MR cerebralcortical segmentation (Part I) a state-of-the-art reviewrdquo PatternAnalysis amp Applications vol 5 no 1 pp 46ndash76 2002

[13] A P Zijdenbos and B M Dawant ldquoBrain segmentation andwhite matter lesion detection in MR imagesrdquo Critical Reviewsin Biomedical Engineering vol 22 no 5-6 pp 401ndash465 1994

[14] K Price ldquoAnything you can do I can do better (No you canrsquot)rdquoComputer Vision Graphics and Image Processing vol 36 no 2-3pp 387ndash391 1986

[15] R de Boer H A Vrooman M A Ikram et al ldquoAccuracy andreproducibility study of automatic MRI brain tissue segmenta-tion methodsrdquo NeuroImage vol 51 no 3 pp 1047ndash1056 2010

[16] J de Bresser M P Portegies A Leemans G J Biessels L JKappelle andM A Viergever ldquoA comparison ofMR based seg-mentation methods for measuring brain atrophy progressionrdquoNeuroImage vol 54 no 2 pp 760ndash768 2011

Computational Intelligence and Neuroscience 15

[17] M B Cuadra L Cammoun T Butz O Cuisenaire and J-P Thiran ldquoComparison and validation of tissue modelizationand statistical classification methods in T1-weighted MR brainimagesrdquo IEEE Transactions on Medical Imaging vol 24 no 12pp 1548ndash1565 2005

[18] F Klauschen A Goldman V Barra A Meyer-Lindenbergand A Lundervold ldquoEvaluation of automated brain MR imagesegmentation and volumetrymethodsrdquoHuman BrainMappingvol 30 no 4 pp 1310ndash1327 2009

[19] H Zaidi T Ruest F Schoenahl and M-L Montandon ldquoCom-parative assessment of statistical brain MR image segmentationalgorithms and their impact on partial volume correction inPETrdquo NeuroImage vol 32 no 4 pp 1591ndash1607 2006

[20] J A Frazier S M Hodge J L Breeze et al ldquoDiagnostic and sexeffects on limbic volumes in early-onset bipolar disorder andschizophreniardquo Schizophrenia Bulletin vol 34 no 1 pp 37ndash462008

[21] A Klein and J Tourville ldquo101 labeled brain images and aconsistent human cortical labeling protocolrdquo Frontiers in Neu-roscience vol 6 article 171 2012

[22] B Van Ginneken T Heimann and M Styner ldquo3D segmen-tation in the clinic a grand challengerdquo in Proceedings of the10th International Conference on Medical Image Computingand Computer Assisted Intervention (MICCAI rsquo07) pp 7ndash15Brisbane Australia October-November 2007

[23] A M Dale B Fischl and M I Sereno ldquoCortical surface-based analysis I Segmentation and surface reconstructionrdquoNeuroImage vol 9 no 2 pp 179ndash194 1999

[24] B Fischl M I Sereno and A M Dale ldquoCortical surface-basedanalysis II inflation flattening and a surface-based coordinatesystemrdquo NeuroImage vol 9 no 2 pp 195ndash207 1999

[25] M Jenkinson C F Beckmann T E J BehrensMWWoolrichand S M Smith ldquoFSLrdquoNeuroImage vol 62 no 2 pp 782ndash7902012

[26] J Ashburner and K J Friston ldquoUnified segmentationrdquo Neu-roImage vol 26 no 3 pp 839ndash851 2005

[27] Y D Reijmer A Leemans M Brundel L J Kappelle and GJ Biessels ldquoDisruption of the cerebral white matter network isrelated to slowing of information processing speed in patientswith type 2 diabetesrdquoDiabetes vol 62 no 6 pp 2112ndash2115 2013

[28] S Klein M Staring K Murphy M A Viergever and J P WPluim ldquoElastix a toolbox for intensity-based medical imageregistrationrdquo IEEE Transactions onMedical Imaging vol 29 no1 pp 196ndash205 2010

[29] K J Friston J T Ashburner S J Kiebel T E Nichols and WD Penny Eds Statistical Parametric Mapping The Analysis ofFunctional Brain Images Academic Press 2011

[30] F Ritter T Boskamp A Homeyer et al ldquoMedical imageanalysis a visual approachrdquo IEEE Pulse vol 2 no 6 pp 60ndash702011

[31] L R Dice ldquoMeasures of the amount of ecologic associationbetween speciesrdquo Ecology vol 26 no 3 pp 297ndash302 1945

[32] D P Huttenlocher G A Klanderman and W J RucklidgeldquoComparing images using the Hausdorff distancerdquo IEEE Trans-actions on Pattern Analysis and Machine Intelligence vol 15 no9 pp 850ndash863 1993

[33] A Alansary A Soliman F Khalifa et al ldquoMAP-based frame-work for segmentation of MR brain images based on visualappearance and prior shaperdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal September 2013

[34] A Jog S Roy J L Prince and A Carass ldquoMR brain segmenta-tion using decision treesrdquo in Proceedings of the MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal Nagoya Japan September 2013

[35] R Katyal S Paneri and M Kuse ldquoGaussian intensity modelwith neighborhood cues for fluid-tissue categorization ofmulti-sequence MR brain imagesrdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal Nagoya Japan September 2013

[36] QMahmoodMAlipoor A Chodorowski AMehnert andMPersson ldquoMultimodal MR brain segmentation using Bayesian-based adaptive mean-shift (BAMS)rdquo in Proceedings of theMICCAI WorkshopsmdashThe MICCAI Grand Challenge on MRBrain Image Segmentation (MRBrainS rsquo13) The Midas JournalSeptember 2013

[37] A van Opbroek F van der Lijn andM de Bruijne ldquoAutomatedbrain-tissue segmentation by multi-feature SVM classificationrdquoin Proceedings of the MICCAI WorkshopsmdashThe MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal September 2013

[38] S Pereira J Festa J A Mariz N Sousa and C A SilvaldquoAutomatic brain tissue segmentation of multi-sequence MRimages using random decision forestsrdquo in Proceedings of theMICCAI Grand Challenge on MR Brain Image Segmentation(MRBrainS rsquo13) The Midas Journal Nagoya Japan September2013

[39] MRajchl J Baxter J Yuan T Peters andAKhan ldquoMulti-atlas-based segmentation with hierarchical max-flowrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[40] D Sarikaya L Zhao and J J Corso ldquoMulti-atlas brain MRIsegmentation with multiway cutrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[41] H Vrooman F van der Lijn and W Niessen ldquoAuto-kNNbrain tissue segmentation using automatically trained k-nearest-neighbor classificationrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[42] S Vyas P Burlina D Kleissas and R Mukherjee ldquoAutomatedwalks using machine learning for segmentationrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[43] C Wang and O Smedby ldquoFully automatic brain segmentationusing model-guided level sets and skeleton-based modelsrdquo inProceedings of theMICCAI Grand Challenge onMRBrain ImageSegmentation (MRBrainS rsquo13) The Midas Journal NagoyaJapan September 2013

[44] C Cortes and V Vapnik ldquoSupport-vector networksrdquo MachineLearning vol 20 no 3 pp 273ndash297 1995

[45] H A Vrooman C A Cocosco F van der Lijn et al ldquoMulti-spectral brain tissue segmentation using automatically trainedk-Nearest-Neighbor classificationrdquo NeuroImage vol 37 no 1pp 71ndash81 2007

[46] C Wang H Frimmel and O Smedby ldquoFast level-set basedimage segmentation using coherent propagationrdquo MedicalPhysics vol 41 no 7 Article ID 073501 2014

16 Computational Intelligence and Neuroscience

[47] A Jog S Roy A Carass and J L Prince ldquoMagnetic resonanceimage synthesis through patch regressionrdquo in Proceedings of theIEEE 10th International Symposium onBiomedical Imaging (ISBIrsquo13) pp 350ndash353 April 2013

[48] J G Sied A P Zijdenbos and A C Evans ldquoA non-parametricmethod for automatic correction of intensity non-uniformity inMRI datardquo IEEE Transactions on Medical Imaging vol 17 no 1pp 87ndash97 1998

[49] A Carass J Cuzzocreo M B Wheeler P-L Bazin S MResnick and J L Prince ldquoSimple paradigm for extra-cerebraltissue removal algorithm and analysisrdquoNeuroImage vol 56 no4 pp 1982ndash1992 2011

[50] M Modat G R Ridgway Z A Taylor et al ldquoFast free-form deformation using graphics processing unitsrdquo ComputerMethods and Programs in Biomedicine vol 98 no 3 pp 278ndash284 2010

[51] M Rajchl J Yuan J A White et al ldquoInteractive hierarchical-flow segmentation of scar tissue from late-enhancement cardiacmr imagesrdquo IEEE Transactions on Medical Imaging vol 33 no1 pp 159ndash172 2014

[52] QMahmood A Chodorowski AMehnert andM Persson ldquoAnovel Bayesian approach to adaptive mean shift segmentationof brain imagesrdquo in Proceedings of the 25th IEEE InternationalSymposiumonComputer-BasedMedical Systems (CBMS rsquo12) pp1ndash6 June 2012

[53] A El-Baz M Nitzken A Alansary A Soliman and MCasanova ldquoBrain Segmentation Method for Young Childrenand Adultsrdquo US Provisional Patent Application 13100

[54] A Alansary A Soliman M Nitzken et al ldquoAn integratedgeometrical and stochastic approach for accurate infant brainextractionrdquo in Proceedings of the IEEE International Conferenceon Image Processing (ICIP rsquo14) pp 3542ndash3546 October 2014

[55] V Popescu M Battaglini W S Hoogstrate et al ldquoOptimizingparameter choice for FSL-Brain Extraction Tool (BET) on 3DT1 images in multiple sclerosisrdquo NeuroImage vol 61 no 4 pp1484ndash1494 2012

[56] J L Whitwell W R Crum H C Watt and N C FoxldquoNormalization of cerebral volumes by use of intracranial vol-ume implications for longitudinal quantitative MR imagingrdquoAmerican Journal of Neuroradiology vol 22 no 8 pp 1483ndash1489 2001

[57] S M Smith ldquoFast robust automated brain extractionrdquo HumanBrain Mapping vol 17 no 3 pp 143ndash155 2002

[58] D W Shattuck G Prasad M Mirza K L Narr and A WToga ldquoOnline resource for validation of brain segmentationmethodsrdquo NeuroImage vol 45 no 2 pp 431ndash439 2009

[59] I Isgum M J N L Benders B Avants et al ldquoEvaluation ofautomatic neonatal brain segmentation algorithms the Neo-BrainS12 challengerdquo Medical Image Analysis vol 20 no 1 pp135ndash151 2015

[60] P Moeskops M J N L Benders S M Chita et al ldquoAutomaticsegmentation of MR brain images of preterm infants usingsupervised classificationrdquo NeuroImage 2015

[61] P Moeskops M A Viergever M J N L Benders and IIsgum ldquoEvaluation of an automatic brain segmentationmethoddeveloped for neonates on adult MR brain imagesrdquo in MedicalImaging Image Processing vol 9413 of Proceedings of SPIEInternational Society for Optics and Photonics Orlando FlaUSA February 2015

[62] L Wang Y Gao F Shi et al ldquoLINKS learning-based multi-source IntegratioN frameworK for Segmentation of infant brainimagesrdquo NeuroImage vol 108 pp 160ndash172 2015

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience 5

Finally these labels are used by the Random Walker forautomatic segmentation

225 LNMBrains [35] The voxel intensities of all MRIsequences are modelled as a Gaussian distribution for eachlabel The parameters of the Gaussian distributions areevaluated asmaximum likelihood estimates and the posteriorprobability of each label is determined by using Bayesianestimation A feature set consisting of regional intensitytexture spatial location of voxels and the posterior prob-ability estimates is used to classify each voxel into CSFWM GM or background by using a multicategory SVMclassifier

226 MNAB [38] This method uses Random DecisionForests to classify the voxels into GM WM and CSF Itstarts by a skull stripping procedure followed by an intensitynormalization of each MRI sequence Feature extraction isthen performed on the intensities posterior probabilitiesneighborhood statistics tissue atlases and gradient mag-nitude After classification isolated voxels are removed bypostprocessing

227 Narsil [34] This is a model-free algorithm that usesensembles of decision trees [47] to learn the mapping fromimage features to the corresponding tissue label The ensem-bles of decision trees are constructed from correspondingimage patches of the provided T1 and FLAIR scans withmanual segmentations The N3 algorithm [48] was used foradditional inhomogeneity correction and SPECTRE [49] wasused for skull stripping

228 Robarts [39] Multiatlas registration [50] with the T1training images was used to propagate labels to generatesample histograms in a log-likelihood intensity model andprobabilistic shape priors These were employed in a MAPdata term and regularized via computation of a hierar-chical max-flow [51] A brain mask from registration ofthe T1-IR training images was used to obtain the finalresults

229 S2 QM [36] This method [52] is based on Bayesian-based adaptive mean shift and the voxel-weighted 119870-meansalgorithm The former is used to segment the brain into alarge number of clusters or modes The latter is employed toassign these clusters to one of the three components WMGM or CSF

2210 UB VPMLMed [40] This method creates a multiatlasby registering the training images to the subject imageand then propagating the corresponding labels to a fullyconnected graph on the subject image Label fusion thencombines the multiple labels into one label at each voxelwith intensity similarity based weighted voting Finally themethod clusters the graph using multiway cut in order toachieve the final segmentation

2211 UofL BioImaging [33] This is an automated MAP-based method aimed at unsupervised segmentation of differ-ent brain tissues from T1-weighted MRI It is based on theintegration of a probabilistic shape prior a first-order inten-sity model using a Linear Combination of Discrete Gaussians(LCDG) and a second-order appearance model These threefeatures are integrated into a two-level joint Markov-GibbsRandom Field (MGRF) model of T1-MR brain imagesSkull stripping was performed using BET2 [40] followed byan adaptive threshold-based technique to restore the outerborder of the CSF using both T1 andT1-IR this techniquewasnot described in [33] due to a US patent application [53] butis described in [54] This method was applied semiautomat-ically to the MRBrainS test data due to per scan parametertuning

2212 Freeware Packages Next to the methods evaluatedat the workshop we evaluated three commonly used free-ware packages for MR brain image segmentation FreeSurfer(httpsurfernmrmghharvardedu) [23 24] FSL (httpfslfmriboxacukfslfslwiki) [25] and SPM12 (httpwwwfilionuclacukspm) [26] All packages were applied usingthe default settings unless mentioned otherwise FreeSurfer(v530) was applied to the high resolution T1 sequence Themri label2vol tool was used to map the labels on the thickslice T1 that was used for the evaluation FSL (v50) wasdirectly applied to the thick slice T1 and provides both apveseg and a seg file as binary output We evaluated bothof these files The fractional intensity threshold parameterldquo119891rdquo of the BET tool that sets the brainnonbrain intensitythreshold was set according to [55] at 02 (Philips Achieva3T setting) SPM12 was directly applied to the thick sliceT1 sequence as well However it also provides the optionto add multiple MRI sequences Therefore we evaluatedSPM12 not only on the thick slice T1 sequence but addedthe T1-IR and the T2-FLAIR scan as well and tested variouscombinationsThe number of Gaussians was set according tothe SPM manual to two for GM two for WM and two forCSF

23 Statistical Analysis All evaluated methods werecompared to the reference standard In summary ofthe results the mean and standard deviation overall 15 test datasets were calculated per component(GM WM and CSF) and combination of components(brain intracranial volume) and per evaluation measure(Dice 95th-percentile Hausdorff distance and absolutevolume difference) for each of the evaluated methodsBoxplots were created using R version 303 (R projectfor statistical computing (httpwwwr-projectorg))Since white matter lesions should be segmented as whitematter the percentage of white matter lesion voxelssegmented as white matter (sensitivity) was calculatedfor each algorithm over all 15 test datasets to evaluatethe robustness of the segmentation algorithms againstpathology

6 Computational Intelligence and Neuroscience

3 Results

Table 1 presents the final ranking (119903) of the evaluatedmethodsthat participated in the workshop as well as the evalu-ated freeware packages During the workshop team UofLBioImaging ranked first and BIGR2 ranked second with onepoint difference in the overall score 119904 (5) However addingthe results of the freeware packages resulted in an equal scorefor UofL BioImaging and BIGR2 Therefore the standarddeviation rank was taken into account and BIGR2 is rankedfirst with standard deviation rank four and UofL BioImagingis ranked second with standard deviation rank eight Table 1further presents the mean standard deviation and rank foreach evaluation measure (119863119867

95 and AVD) and component

(GM WM and CSF) as well as the brain (WM + GM) andintracranial volume (WM+GM+CSF) Team BIGR2 scoredbest for theGMWM and brain segmentation and teamUofLBioImaging for the CSF segmentation Team Robarts scoredbest for the intracranial volume segmentation The boxplotsfor all evaluation measures and components are shown inFigures 2ndash4 and include the results of the freeware packagesFigure 5 shows an example of the segmentation results atthe height of the basal ganglia (slice 22 of test subject 9)The sensitivity of the algorithms to segment white matterlesions as WM and examples of the segmentation results inthe presence of white matter lesions (slice 31 of test subject3) are shown in Figure 6 Team UB VPML Med scores thehighest sensitivity of white matter lesions segmented as whitematter and is therefore most robust in the presence of thistype of pathology

4 Discussion

In this paper we proposed the MRBrainS challenge onlineevaluation framework to evaluate automatic and semiauto-matic algorithms for segmenting GM WM and CSF on3T multisequence (T1 T1-IR and T2-FLAIR) MRI scans ofthe brain We have evaluated and presented the results ofeleven segmentation algorithms that provide a benchmarkfor algorithms that will use the online evaluation frameworkto evaluate their performance Team UofL BioImaging andBIGR2 have equal overall scores but BIGR2 was rankedfirst based on the standard deviation ranking The evalu-ated methods represent a wide variety of algorithms thatinclude Markov random field models clustering approachesdeformable models and atlas-based approaches and classi-fiers (SVM KNN and decision trees) The presented evalua-tion framework provides an insight into the performance ofthese algorithms in terms of accuracy and robustness Variousfactors influence the choice for a certain method aboveothers We provide three measures that could aid in selectingthe method that is most appropriate for the segmentationgoal at hand a boundarymeasure (95th-percentile Hausdorffdistance 119867

95) an overlap measure (Dice coefficient 119863) and

a volume measure (absolute volume difference AVD) Allthree measures are taken into account for the final rankingof the methods This ranking was designed to get a quickinsight into how the methods perform in comparison to eachother The best overall method is the method that performs

well for all three measures and all three components (GMWM andCSF) However whichmethod to select depends onthe segmentation goal at hand Not all measures are relevantfor all segmentation goals For example if segmentation isused for brain volumetry [4] the overlap (119863) and volume(AVD) measures of the brain and intracranial volume (usedfor normalization [56]) segmentations are important to takeinto account On the other hand if segmentation is usedfor cortical thickness measurements the focus should be onthe gray matter boundary (119867

95) and overlap (119863) measures

Therefore the final ranking should be used to get a first insightinto the overall performance after which the performanceof the measures and components that are most relevant forthe segmentation goal at hand should be considered Besidesaccuracy robustness could also influence the choice for acertain method above others For example team UB VPMLMed shows a high sensitivity score for segmenting whitemat-ter lesions as white matter (Figure 6) and shows a consistentsegmentation performance of gray and white matter over all15 test datasets (Figures 2ndash4) This could be beneficial forsegmenting scans of populations with white matter lesionsbut is less important if the goal is to segment scans ofyoung healthy subjects In the latter case the most accuratesegmentation for gray and white matter (team BIGR2) ismore interesting If a segmentation algorithm is to be usedin clinical practice speed is an important consideration aswell The runtime of the evaluated methods is reported inTable 1 However these runtimes are merely an indicationof the required time since academic software is generallynot optimized for speed and the runtime is measured ondifferent computers andplatformsAnother relevant aspect ofthe evaluation framework is the comparison of multi- versussingle-sequence approaches For example most methodsstruggle with the segmentation of the intracranial volume onthe T1-weighted scan There is no contrast between the CSFand the skull and the contrast between the dura mater andthe CSF is not always sufficient Team Robarts used an atlas-based registration approach on the T1-IR scan (good contrastbetween skull and CSF) to segment the intracranial volumewhich resulted in the best performance for intracranialvolume segmentation (Table 1 Figures 2ndash4) Most methodsadd the T2-FLAIR scan to improve robustness against whitematter lesions (Table 1 Figure 6) Although using only theT1-weighted scan and incorporating prior shape information(team UofL BioImaging) can be very effective also thefreeware packages support this as well Since FreeSurfer isan atlas-based method it uses prior information and is themost robust of all freeware packages to white matter lesionsHowever adding the T2 FLAIR scan to SPM12 increasesrobustness against white matter lesions as well as comparedto applying SPM12 to the T1 scan only (Figure 6) In generalSPM12with the T1 and the T2-FLAIR sequence performswellin comparison to the other freeware packages (Table 1 andFigures 2ndash4) on the thick slice MRI scans Although addingthe T1-IR scan to SPM increases the performance of the CSFand ICV segmentations as compared to using only the T1 andT2-FLAIR sequence it decreases the performance of the GMand WM segmentations Therefore adding all sequences toSPM12 did not result in a better overall performance

Computational Intelligence and Neuroscience 7

Table 1 Results of the 11 evaluated algorithms presented at the workshop and the evaluated freeware packages on the 15 test datasets Thealgorithms are ranked (119903) based on their overall score (119904) by using (5) This score is based on the ranks of the gray matter (GM) white matter(WM) and cerebrospinal fluid (CSF) segmentation and the three evaluated measures Dice coefficient (119863 in ) 95th-percentile Hausdorffdistance (119867

95in mm) and the absolute volume difference (AVD in ) The rank 119903

119898119888denotes the rank based on the mean (120583) over all 15 test

datasets for each measure 119898 (0 119863 1 11986795 and 2 AVD) and component 119888 (0 GM 1 WM 2 CSF 3 brain (WM + GM) and 4 intracranial

volume (ICV = WM + GM + CSF)) Teams BIGR2 and UofL BioImaging and FreeSurfer and Jedi Mind Meld have equal scores based onthe mean (120583) therefore the ranking based on the standard deviation (120590) is taken into account to determine the final rank (BIGR2 120590 rank 4UofL BioImaging 120590 rank 8 FreeSurfer 120590 rank 13 and Jedi MindMeld 120590 rank 17) Columns 2 and 3 present the average runtime 119905 per scan inseconds (s) minutes (m) or hours (h) and the scans (T1 T1-weighted scan 3D T1 3D T1-weighted scan IR T1-weighted inversion recovery(IR) and F T1-weighted FLAIR) that are used for processing

119903 Team 119905 Scans

GM WM CSF

119904

Brain ICV119903001199031011990320119903011199031111990321119903021199031211990322

119903031199031311990323119903041199031411990324

119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590)

1 BIGR2 35m T1 IR F1 2 4 2 2 4 4 5 14

381 2 12 8 4 12

847 19 61 884 24 60 783 32 23 951 27 32 960 39 52(13) (04) (33) (12) (05) (51) (50) (08) (17) (05) (08) (16) (13) (11) (30)

2 UofLBioImaginglowast 6 s T1

5 1 9 4 1 13 2 2 138

2 1 12 5 2 5830 17 86 879 22 87 789 27 97 949 24 39 967 34 18(15) (03) (54) (20) (06) (66) (42) (05) (10) (06) (05) (20) (08) (06) (20)

3 CMIV 3m T1 F6 7 5 5 3 10 3 3 8

505 7 2 4 3 11

824 27 68 877 24 73 786 30 14 945 38 26 968 38 49(14) (04) (40) (16) (04) (38) (31) (04) (59) (05) (11) (22) (08) (13) (23)

4 UB VPMLMed 30m T1 IR F

4 4 2 1 5 7 8 13 1761

4 4 1 10 11 15833 21 59 886 27 71 748 43 31 946 28 24 948 66 77(13) (03) (53) (17) (04) (38) (71) (17) (19) (06) (04) (18) (20) (20) (41)

5 Bigr neuro 2 h T1 F7 13 3 6 6 9 6 4 10

647 10 10 7 5 8

815 37 59 873 30 73 782 32 16 940 46 36 963 39 35(17) (09) (42) (14) (04) (38) (47) (06) (14) (08) (14) (24) (12) (09) (27)

6 Robarts 16m 3D T1 IR11 3 15 8 8 6 1 1 13

6616 3 18 1 1 1

797 20 98 862 31 71 803 27 20 931 28 79 979 26 09(24) (01) (73) (13) (04) (62) (41) (05) (13) (16) (05) (36) (03) (04) (07)

7 Narsil 2m T1 F3 5 1 7 11 2 17 18 7

713 5 3 16 18 9

835 23 55 871 33 58 666 133 14 948 29 29 925 24 37(18) (04) (44) (13) (09) (53) (24) (54) (95) (05) (05) (20) (05) (89) (17)

8 SPM T1 F 3m T1 F8 9 16 9 7 1 9 14 2

759 14 14 6 14 4

812 29 10 860 30 52 741 46 10 939 58 53 966 82 15(22) (03) (85) (15) (01) (38) (34) (06) (47) (10) (22) (38) (02) (30) (10)

9 SPM T1 IR 3m T1 IR12 11 7 16 12 5 5 10 3

818 11 7 2 8 2

794 30 72 835 36 63 783 40 10 939 46 34 977 65 10(21) (04) (63) (21) (03) (46) (38) (06) (57) (08) (12) (28) (02) (13) (08)

10 MNAB 15m T1 IR F2 8 12 3 4 11 15 15 16

866 9 11 15 12 17

839 28 91 880 27 78 681 49 29 945 45 38 925 71 97(21) (09) (65) (12) (08) (40) (40) (22) (21) (10) (20) (32) (11) (42) (47)

11 SPM T1 3m T19 10 6 11 10 3 11 16 15

9110 8 6 9 13 13

803 30 69 856 31 60 707 53 23 939 44 32 953 81 55(24) (05) (68) (17) (01) (41) (38) (15) (157) (09) (16) (29) (09) (37) (37)

12 FSL Seg 10m T113 16 10 10 13 14 12 6 5

9913 13 4 12 6 6

787 43 86 860 37 115 699 34 12 933 55 30 942 53 34(22) (12) (63) (26) (08) (63) (28) (02) (103) (08) (14) (15) (08) (11) (15)

8 Computational Intelligence and Neuroscience

Table 1 Continued

119903 Team 119905 Scans

GM WM CSF

119904

Brain ICV119903001199031011990320119903011199031111990321119903021199031211990322

119903031199031311990323119903041199031411990324

119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590)

13 SPM T1 IR F 4m T1 IR F10 12 18 15 14 12 7 11 6

10512 15 13 3 15 3

801 30 139 836 38 84 769 41 12 936 59 51 977 82 12(24) (02) (96) (21) (05) (52) (31) (05) (60) (11) (18) (36) (02) (18) (09)

14 FSL PVSeg 10m T115 15 8 13 15 17 13 7 4

10711 16 9 13 7 7

777 43 84 848 38 197 695 34 11 936 61 35 942 53 34(26) (13) (65) (32) (09) (10) (22) (03) (58) (09) (16) (30) (08) (11) (15)

15 FreeSurfer 1 h 3D T116 6 17 12 9 8 18 12 18

11617 6 16 17 10 18

774 23 121 852 31 72 658 43 50 923 36 63 924 66 10(20) (06) (60) (22) (05) (46) (37) (06) (196) (08) (06) (34) (09) (04) (45)

16 Jedi MindMeld 27 s T1 IR F

14 14 11 17 16 15 10 8 11116

15 12 8 11 16 14778 39 89 806 45 137 733 37 18 932 54 35 943 20 68(59) (28) (76) (96) (36) (12) (35) (08) (90) (17) (48) (30) (17) (37) (31)

17 S2 QM 15 h T1 IR F17 17 13 14 17 18 16 9 9

13014 17 15 14 9 10

764 55 93 839 49 248 679 38 14 933 70 55 937 65 40(34) (30) (64) (34) (32) (10) (23) (06) (99) (14) (35) (45) (11) (14) (22)

18 LNMBrains 5m T1 IR F18 18 14 18 18 16 14 17 12

14518 18 17 18 17 16

728 68 95 783 68 153 688 77 20 885 86 74 892 204 79(53) (23) (76) (64) (35) (13) (66) (24) (13) (45) (28) (77) (48) (37) (83)

lowastSemiautomatic due to per scan parameter tuning

Besides the advantages of the MRBrainS evaluationframework there are some limitations that should be takeninto account The T1-weighted IR and the T2-weightedFLAIR scan were acquired with a lower resolution (096times 096 times 300mm3) than the 3D T1-weighted scan (10 times10 times 10mm3) To be able to provide a registered multise-quence dataset the 3D T1-weighted scan was registered tothe T2-weighted FLAIR scan and downsampled to 096 times096 times 300mm3 The reference standard is therefore onlyavailable for this resolution The decreased performance ofthe FreeSurfer GM segmentation as compared to the otherfreeware packages might be due to the fact that we evaluateon the thick slice T1 sequence instead of the high resolutionT1 Performing the manual segmentations to provide thereference standard is very laborsome and time consumingInstead of letting multiple observers manually segment theMRI datasets or letting one observer manually segmentthe MRI datasets twice much time and effort was spenton creating one reference standard that was as accurate aspossible Therefore we were not able to determine the inter-or intraobserver variability Finally we acknowledge that ourevaluation framework is limited to evaluating the accuracyand robustness over 15 datasets for segmenting GM WMand CSF on 3T MRI scans acquired on a Philips scanner ofa specific group of elderly subjects Many factors influencesegmentation algorithm performance such as the type ofscanner (vendor field strength) the acquisition protocolthe available MRI sequences and the type of subjects

Participating algorithmsmight have been designed for differ-ent types of MRI scans Therefore the five provided trainingdatasets are important for participants to be able to traintheir algorithms on the provided data Some algorithms aredesigned to segment only some components such as onlyGM and WM instead of all three components and usefreely available software such as the brain extraction tool[57] to segment the outer border of the CSF (intracranialvolume) We have chosen to base the final ranking on allthree components but it is therefore important to assess notonly the final ranking but the performance of the individualcomponents as well

Despite these limitations the MRBrainS evaluationframework provides an objective and direct comparison ofsegmentation algorithms The reference standard of the testdata is unknown to the participants the same evaluationmeasures are used for all evaluated algorithms and partici-pants apply their own algorithms to the provided data

In comparison to the online validation engine proposedby Shattuck et al [58] the MRBrainS evaluation frameworkuses 3T MRI data instead of 15T MRI data and evaluates notonly brain versus nonbrain segmentation but also segmenta-tion of gray matter white matter cerebrospinal fluid brainand intracranial volume The availability of many differenttypes of evaluation frameworks will aid in the developmentof more generic and robust algorithms For example inthe NEATBrainS (httpneatbrains15isiuunl) challengeresearchers were challenged to apply their algorithms to data

Computational Intelligence and Neuroscience 9

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

65

70

75

80

85

GM Dice

Dic

e (

)

(a)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

55

60

65

70

75

80

85

90

WM Dice

Dic

e (

)

(b)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

55

60

65

70

75

80

85CSF Dice

Dic

e (

)

(c)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

80

85

90

95

Brain (WM + GM) Dice

Dic

e (

)

(d)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

80

85

90

95

Dic

e (

)

Intracranial volume (WM + GM + CSF) Dice

(e)

Figure 2 Boxplots presenting the evaluation results for the Dice coefficient (1) of the gray matter (GM) white matter (WM) cerebrospinalfluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all 15 test datasets

10 Computational Intelligence and Neuroscience

2468

10121416

GM H-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(a)

2468

10121416

WMH-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(b)

5

10

15

20CSF H-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(c)

5

10

15

20H

-95

(mm

)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Brain (WM + GM) H-95

(d)

5

10

15

20

25

30

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Intracranial volume (WM + GM + CSF) H-95

(e)

Figure 3 Boxplots presenting the evaluation results for the 95th-percentile Hausdorff distance (3) of the gray matter (GM) white matter(WM) cerebrospinal fluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all15 test datasets

Computational Intelligence and Neuroscience 11

0

10

20

30

GM AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(a)

0

10

20

30

40WM AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(b)

0

20

40

60

80

CSF AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(c)

0

5

10

15

20

25

30AV

D (

)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Brain (WM + GM) AVD

(d)

0

5

10

15

20

25

30

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Intracranial volume (WM + GM + CSF) AVD

(e)

Figure 4 Boxplots presenting the evaluation results for the absolute volume difference (4) of the gray matter (GM) white matter (WM)cerebrospinal fluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all 15 testdatasets

12 Computational Intelligence and Neuroscience

T2

UofL

N

FS

S2

‐FLAIR

BioImaging

Narsil

L‐Seg

2_QM

T1‐IR

CMIV

SPM12_T1

SPM12_T1_

LNMBr

_FLAIR

_IR_FLAIR

ains

T1

UB VPML Me

SPM12_T1_ I

FSL‐PVSeg

R

d B

R

F

eference

igr_neuro

MNAB

reeSurfer

BIG

Rob

SPM1

Jedi Mi

R2

arts

2_T1

nd Meld

Figure 5 Illustration of the segmentation results at the height of the basal ganglia (test subject 9 slice 22) The basal ganglia should besegmented as gray matter The first three images show the three MRI sequences The fourth image (reference) is the manually segmentedreference standard (yellow white matter light blue gray matter and dark blue cerebrospinal fluid) The results of the participatingsegmentation algorithms are ordered from the best overall performance (BIGR2) to the worst overall performance (LNMBrains) Majordifferences are present in the segmentation of the basal ganglia and the outer border of the cerebrospinal fluid The arrows indicate examplelocations where the segmentation results differ from the ground truth

Computational Intelligence and Neuroscience 13

T2

UofL BioIm

Nars

FSL‐S

S2_Q

‐FLAIR

aging (860)

il (892)

eg (453)

M (565)

T1‐IR

CMIV (90

SPM12_T1_F

SPM12_T1_IR

LNMBrains

5) UB

(831) SP

_F (815) F

(286)

T1

VPML Med (9

M12_T1_IR (60

SL‐PVSeg (525

R

10) Bigr_

5) MN

) FreeS

eference

neuro (896)

AB (889)

urfer (846)

BIGR2

Robarts

SPM12_T

Jedi Mind M

(852)

(810)

1 (683)

eld (769)

Figure 6 Illustration of the segmentation results in the presence of whitematter lesions (test subject 3 slice 31)Whitematter lesions (WMLs)should be labeled asWM the sensitivity (percentage ofWML voxels is labeled asWM) over all 15 datasets is presented between brackets afterthe team nameThe first three images show the threeMRI sequences T2-weighted fluid attenuated inversion recovery T1-weighted inversionrecovery and T1-weighted scanThe fourth image (reference) is themanually segmented reference standard (red whitematter lesions yellowwhite matter light blue gray matter and dark blue cerebrospinal fluid)The results of the segmentation algorithms are ordered from the bestoverall performance (BIGR2) to the worst overall performance (LNMBrains)The arrows indicate example locations where the segmentationresults differ from the ground truth

14 Computational Intelligence and Neuroscience

fromboth theMRBrainS and theNeoBrainS [59] (brain tissuesegmentation in neonates) challenge Two methods [60ndash62]specifically designed for neonatal brain tissue segmentationshowed a high performance for tissue segmentation on theMRBrainS data Applying algorithms to different types of datahas the potential to lead to new insights and more robustalgorithms The MRBrainS evaluation framework remainsopen for new contributions At the time of writing 21teams had submitted their results on the MRBrainS website(httpmrbrains13isiuunlresultsphp)

5 Conclusion

The MRBrainS challenge online evaluation framework pro-vides direct and objective comparison of automatic andsemiautomatic methods to segment GM WM CSF brainand ICV on 3T multisequence MRI data The first elevenparticipating methods are evaluated and presented in thispaper as well as three commonly used freeware packages(FreeSurfer FSL and SPM12) They provide a benchmarkfor future contributions to the framework The final rankingprovides a quick insight into the overall performance of theevaluated methods in comparison to each other whereasthe individual evaluation measures (Dice 95th-percentileHausdorff distance and absolute volume difference) percomponent (GM WM CSF brain and ICV) can aid inselecting the best method for a specific segmentation goal

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was financially supported by IMDI Grant104002002 (Brainbox) fromZonMw the Netherlands Organ-isation for Health Research and Development within kindsponsoring by Philips theUniversityMedical CenterUtrechtand Eindhoven University of TechnologyThe authors wouldlike to acknowledge the following members of the UtrechtVascular Cognitive Impairment Study Group who were notincluded as coauthors of this paper but were involved in therecruitment of study participants and MRI acquisition at theUMC Utrecht (in alphabetical order by department) E vanden Berg M Brundel S Heringa and L J Kappelle of theDepartment of Neurology P R Luijten and W P Th MMali of the Department of Radiology and A Algra and GE H M Rutten of the Julius Center for Health Sciences andPrimary CareThe research of Geert Jan Biessels and the VCIgroupwas financially supported byVIDIGrant 91711384 fromZonMw and by Grant 2010T073 of the Netherlands HeartFoundation The research of Jeroen de Bresser is financiallysupported by a research talent fellowship of the Univer-sity Medical Center Utrecht (Netherlands) The research ofAnnegreet vanOpbroek andMarleen de Bruijne is financiallysupported by a research grant from NWO (the NetherlandsOrganisation for Scientific Research) The authors would

like to acknowledge MeVis Medical Solutions AG (BremenGermany) for providing MeVisLab Duygu Sarikaya andLiang Zhao acknowledge their Advisor Professor JasonCorsofor his guidance Duygu Sarikaya is supported by NIH 1 R21CA160825-01 and Liang Zhao is partially supported by theChina Scholarship Council (CSC)

References

[1] I Driscoll C Davatzikos Y An et al ldquoLongitudinal pattern ofregional brain volume change differentiates normal aging fromMCIrdquo Neurology vol 72 no 22 pp 1906ndash1913 2009

[2] K Franke and C Gaser ldquoLongitudinal changes in individualBrainAGE in healthy aging mild cognitive impairment andAlzheimerrsquos diseaserdquoGeroPsych vol 25 no 4 pp 235ndash245 2012

[3] MA IkramH A VroomanMWVernooij et al ldquoBrain tissuevolumes in the general elderly populationThe Rotterdam ScanStudyrdquo Neurobiology of Aging vol 29 no 6 pp 882ndash890 2008

[4] A Giorgio and N de Stefano ldquoClinical use of brain volumetryrdquoJournal of Magnetic Resonance Imaging vol 37 no 1 pp 1ndash142013

[5] M W Vannier R L Butterfield D Jordan W A Murphy RG Levitt and M Gado ldquoMultispectral analysis of magneticresonance imagesrdquo Radiology vol 154 no 1 pp 221ndash224 1985

[6] M A Balafar A R Ramli M I Saripan and S MashohorldquoReview of brain MRI image segmentation methodsrdquo ArtificialIntelligence Review vol 33 no 3 pp 261ndash274 2010

[7] J C Bezdek L O Hall and L P Clarke ldquoReview of MR imagesegmentation techniques using pattern recognitionrdquo MedicalPhysics vol 20 no 4 pp 1033ndash1048 1993

[8] L P Clarke R P Velthuizen M A Camacho et al ldquoMRIsegmentation methods and applicationsrdquo Magnetic ResonanceImaging vol 13 no 3 pp 343ndash368 1995

[9] J S Duncan X Papademetris J Yang M Jackowski X Zengand L H Staib ldquoGeometric strategies for neuroanatomic ana-lysis from MRIrdquo NeuroImage vol 23 supplement 1 pp S34ndashS45 2004

[10] D L PhamC Xu and J L Prince ldquoCurrentmethods inmedicalimage segmentationrdquoAnnual Review of Biomedical Engineeringvol 2 no 2000 pp 315ndash337 2000

[11] N Saeed ldquoMagnetic resonance image segmentation usingpattern recognition and applied to image registration andquantitationrdquo NMR in Biomedicine vol 11 no 4-5 pp 157ndash1671998

[12] J S Suri S Singh and L Reden ldquoComputer vision andpattern recognition techniques for 2-D and 3-D MR cerebralcortical segmentation (Part I) a state-of-the-art reviewrdquo PatternAnalysis amp Applications vol 5 no 1 pp 46ndash76 2002

[13] A P Zijdenbos and B M Dawant ldquoBrain segmentation andwhite matter lesion detection in MR imagesrdquo Critical Reviewsin Biomedical Engineering vol 22 no 5-6 pp 401ndash465 1994

[14] K Price ldquoAnything you can do I can do better (No you canrsquot)rdquoComputer Vision Graphics and Image Processing vol 36 no 2-3pp 387ndash391 1986

[15] R de Boer H A Vrooman M A Ikram et al ldquoAccuracy andreproducibility study of automatic MRI brain tissue segmenta-tion methodsrdquo NeuroImage vol 51 no 3 pp 1047ndash1056 2010

[16] J de Bresser M P Portegies A Leemans G J Biessels L JKappelle andM A Viergever ldquoA comparison ofMR based seg-mentation methods for measuring brain atrophy progressionrdquoNeuroImage vol 54 no 2 pp 760ndash768 2011

Computational Intelligence and Neuroscience 15

[17] M B Cuadra L Cammoun T Butz O Cuisenaire and J-P Thiran ldquoComparison and validation of tissue modelizationand statistical classification methods in T1-weighted MR brainimagesrdquo IEEE Transactions on Medical Imaging vol 24 no 12pp 1548ndash1565 2005

[18] F Klauschen A Goldman V Barra A Meyer-Lindenbergand A Lundervold ldquoEvaluation of automated brain MR imagesegmentation and volumetrymethodsrdquoHuman BrainMappingvol 30 no 4 pp 1310ndash1327 2009

[19] H Zaidi T Ruest F Schoenahl and M-L Montandon ldquoCom-parative assessment of statistical brain MR image segmentationalgorithms and their impact on partial volume correction inPETrdquo NeuroImage vol 32 no 4 pp 1591ndash1607 2006

[20] J A Frazier S M Hodge J L Breeze et al ldquoDiagnostic and sexeffects on limbic volumes in early-onset bipolar disorder andschizophreniardquo Schizophrenia Bulletin vol 34 no 1 pp 37ndash462008

[21] A Klein and J Tourville ldquo101 labeled brain images and aconsistent human cortical labeling protocolrdquo Frontiers in Neu-roscience vol 6 article 171 2012

[22] B Van Ginneken T Heimann and M Styner ldquo3D segmen-tation in the clinic a grand challengerdquo in Proceedings of the10th International Conference on Medical Image Computingand Computer Assisted Intervention (MICCAI rsquo07) pp 7ndash15Brisbane Australia October-November 2007

[23] A M Dale B Fischl and M I Sereno ldquoCortical surface-based analysis I Segmentation and surface reconstructionrdquoNeuroImage vol 9 no 2 pp 179ndash194 1999

[24] B Fischl M I Sereno and A M Dale ldquoCortical surface-basedanalysis II inflation flattening and a surface-based coordinatesystemrdquo NeuroImage vol 9 no 2 pp 195ndash207 1999

[25] M Jenkinson C F Beckmann T E J BehrensMWWoolrichand S M Smith ldquoFSLrdquoNeuroImage vol 62 no 2 pp 782ndash7902012

[26] J Ashburner and K J Friston ldquoUnified segmentationrdquo Neu-roImage vol 26 no 3 pp 839ndash851 2005

[27] Y D Reijmer A Leemans M Brundel L J Kappelle and GJ Biessels ldquoDisruption of the cerebral white matter network isrelated to slowing of information processing speed in patientswith type 2 diabetesrdquoDiabetes vol 62 no 6 pp 2112ndash2115 2013

[28] S Klein M Staring K Murphy M A Viergever and J P WPluim ldquoElastix a toolbox for intensity-based medical imageregistrationrdquo IEEE Transactions onMedical Imaging vol 29 no1 pp 196ndash205 2010

[29] K J Friston J T Ashburner S J Kiebel T E Nichols and WD Penny Eds Statistical Parametric Mapping The Analysis ofFunctional Brain Images Academic Press 2011

[30] F Ritter T Boskamp A Homeyer et al ldquoMedical imageanalysis a visual approachrdquo IEEE Pulse vol 2 no 6 pp 60ndash702011

[31] L R Dice ldquoMeasures of the amount of ecologic associationbetween speciesrdquo Ecology vol 26 no 3 pp 297ndash302 1945

[32] D P Huttenlocher G A Klanderman and W J RucklidgeldquoComparing images using the Hausdorff distancerdquo IEEE Trans-actions on Pattern Analysis and Machine Intelligence vol 15 no9 pp 850ndash863 1993

[33] A Alansary A Soliman F Khalifa et al ldquoMAP-based frame-work for segmentation of MR brain images based on visualappearance and prior shaperdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal September 2013

[34] A Jog S Roy J L Prince and A Carass ldquoMR brain segmenta-tion using decision treesrdquo in Proceedings of the MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal Nagoya Japan September 2013

[35] R Katyal S Paneri and M Kuse ldquoGaussian intensity modelwith neighborhood cues for fluid-tissue categorization ofmulti-sequence MR brain imagesrdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal Nagoya Japan September 2013

[36] QMahmoodMAlipoor A Chodorowski AMehnert andMPersson ldquoMultimodal MR brain segmentation using Bayesian-based adaptive mean-shift (BAMS)rdquo in Proceedings of theMICCAI WorkshopsmdashThe MICCAI Grand Challenge on MRBrain Image Segmentation (MRBrainS rsquo13) The Midas JournalSeptember 2013

[37] A van Opbroek F van der Lijn andM de Bruijne ldquoAutomatedbrain-tissue segmentation by multi-feature SVM classificationrdquoin Proceedings of the MICCAI WorkshopsmdashThe MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal September 2013

[38] S Pereira J Festa J A Mariz N Sousa and C A SilvaldquoAutomatic brain tissue segmentation of multi-sequence MRimages using random decision forestsrdquo in Proceedings of theMICCAI Grand Challenge on MR Brain Image Segmentation(MRBrainS rsquo13) The Midas Journal Nagoya Japan September2013

[39] MRajchl J Baxter J Yuan T Peters andAKhan ldquoMulti-atlas-based segmentation with hierarchical max-flowrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[40] D Sarikaya L Zhao and J J Corso ldquoMulti-atlas brain MRIsegmentation with multiway cutrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[41] H Vrooman F van der Lijn and W Niessen ldquoAuto-kNNbrain tissue segmentation using automatically trained k-nearest-neighbor classificationrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[42] S Vyas P Burlina D Kleissas and R Mukherjee ldquoAutomatedwalks using machine learning for segmentationrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[43] C Wang and O Smedby ldquoFully automatic brain segmentationusing model-guided level sets and skeleton-based modelsrdquo inProceedings of theMICCAI Grand Challenge onMRBrain ImageSegmentation (MRBrainS rsquo13) The Midas Journal NagoyaJapan September 2013

[44] C Cortes and V Vapnik ldquoSupport-vector networksrdquo MachineLearning vol 20 no 3 pp 273ndash297 1995

[45] H A Vrooman C A Cocosco F van der Lijn et al ldquoMulti-spectral brain tissue segmentation using automatically trainedk-Nearest-Neighbor classificationrdquo NeuroImage vol 37 no 1pp 71ndash81 2007

[46] C Wang H Frimmel and O Smedby ldquoFast level-set basedimage segmentation using coherent propagationrdquo MedicalPhysics vol 41 no 7 Article ID 073501 2014

16 Computational Intelligence and Neuroscience

[47] A Jog S Roy A Carass and J L Prince ldquoMagnetic resonanceimage synthesis through patch regressionrdquo in Proceedings of theIEEE 10th International Symposium onBiomedical Imaging (ISBIrsquo13) pp 350ndash353 April 2013

[48] J G Sied A P Zijdenbos and A C Evans ldquoA non-parametricmethod for automatic correction of intensity non-uniformity inMRI datardquo IEEE Transactions on Medical Imaging vol 17 no 1pp 87ndash97 1998

[49] A Carass J Cuzzocreo M B Wheeler P-L Bazin S MResnick and J L Prince ldquoSimple paradigm for extra-cerebraltissue removal algorithm and analysisrdquoNeuroImage vol 56 no4 pp 1982ndash1992 2011

[50] M Modat G R Ridgway Z A Taylor et al ldquoFast free-form deformation using graphics processing unitsrdquo ComputerMethods and Programs in Biomedicine vol 98 no 3 pp 278ndash284 2010

[51] M Rajchl J Yuan J A White et al ldquoInteractive hierarchical-flow segmentation of scar tissue from late-enhancement cardiacmr imagesrdquo IEEE Transactions on Medical Imaging vol 33 no1 pp 159ndash172 2014

[52] QMahmood A Chodorowski AMehnert andM Persson ldquoAnovel Bayesian approach to adaptive mean shift segmentationof brain imagesrdquo in Proceedings of the 25th IEEE InternationalSymposiumonComputer-BasedMedical Systems (CBMS rsquo12) pp1ndash6 June 2012

[53] A El-Baz M Nitzken A Alansary A Soliman and MCasanova ldquoBrain Segmentation Method for Young Childrenand Adultsrdquo US Provisional Patent Application 13100

[54] A Alansary A Soliman M Nitzken et al ldquoAn integratedgeometrical and stochastic approach for accurate infant brainextractionrdquo in Proceedings of the IEEE International Conferenceon Image Processing (ICIP rsquo14) pp 3542ndash3546 October 2014

[55] V Popescu M Battaglini W S Hoogstrate et al ldquoOptimizingparameter choice for FSL-Brain Extraction Tool (BET) on 3DT1 images in multiple sclerosisrdquo NeuroImage vol 61 no 4 pp1484ndash1494 2012

[56] J L Whitwell W R Crum H C Watt and N C FoxldquoNormalization of cerebral volumes by use of intracranial vol-ume implications for longitudinal quantitative MR imagingrdquoAmerican Journal of Neuroradiology vol 22 no 8 pp 1483ndash1489 2001

[57] S M Smith ldquoFast robust automated brain extractionrdquo HumanBrain Mapping vol 17 no 3 pp 143ndash155 2002

[58] D W Shattuck G Prasad M Mirza K L Narr and A WToga ldquoOnline resource for validation of brain segmentationmethodsrdquo NeuroImage vol 45 no 2 pp 431ndash439 2009

[59] I Isgum M J N L Benders B Avants et al ldquoEvaluation ofautomatic neonatal brain segmentation algorithms the Neo-BrainS12 challengerdquo Medical Image Analysis vol 20 no 1 pp135ndash151 2015

[60] P Moeskops M J N L Benders S M Chita et al ldquoAutomaticsegmentation of MR brain images of preterm infants usingsupervised classificationrdquo NeuroImage 2015

[61] P Moeskops M A Viergever M J N L Benders and IIsgum ldquoEvaluation of an automatic brain segmentationmethoddeveloped for neonates on adult MR brain imagesrdquo in MedicalImaging Image Processing vol 9413 of Proceedings of SPIEInternational Society for Optics and Photonics Orlando FlaUSA February 2015

[62] L Wang Y Gao F Shi et al ldquoLINKS learning-based multi-source IntegratioN frameworK for Segmentation of infant brainimagesrdquo NeuroImage vol 108 pp 160ndash172 2015

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

6 Computational Intelligence and Neuroscience

3 Results

Table 1 presents the final ranking (119903) of the evaluatedmethodsthat participated in the workshop as well as the evalu-ated freeware packages During the workshop team UofLBioImaging ranked first and BIGR2 ranked second with onepoint difference in the overall score 119904 (5) However addingthe results of the freeware packages resulted in an equal scorefor UofL BioImaging and BIGR2 Therefore the standarddeviation rank was taken into account and BIGR2 is rankedfirst with standard deviation rank four and UofL BioImagingis ranked second with standard deviation rank eight Table 1further presents the mean standard deviation and rank foreach evaluation measure (119863119867

95 and AVD) and component

(GM WM and CSF) as well as the brain (WM + GM) andintracranial volume (WM+GM+CSF) Team BIGR2 scoredbest for theGMWM and brain segmentation and teamUofLBioImaging for the CSF segmentation Team Robarts scoredbest for the intracranial volume segmentation The boxplotsfor all evaluation measures and components are shown inFigures 2ndash4 and include the results of the freeware packagesFigure 5 shows an example of the segmentation results atthe height of the basal ganglia (slice 22 of test subject 9)The sensitivity of the algorithms to segment white matterlesions as WM and examples of the segmentation results inthe presence of white matter lesions (slice 31 of test subject3) are shown in Figure 6 Team UB VPML Med scores thehighest sensitivity of white matter lesions segmented as whitematter and is therefore most robust in the presence of thistype of pathology

4 Discussion

In this paper we proposed the MRBrainS challenge onlineevaluation framework to evaluate automatic and semiauto-matic algorithms for segmenting GM WM and CSF on3T multisequence (T1 T1-IR and T2-FLAIR) MRI scans ofthe brain We have evaluated and presented the results ofeleven segmentation algorithms that provide a benchmarkfor algorithms that will use the online evaluation frameworkto evaluate their performance Team UofL BioImaging andBIGR2 have equal overall scores but BIGR2 was rankedfirst based on the standard deviation ranking The evalu-ated methods represent a wide variety of algorithms thatinclude Markov random field models clustering approachesdeformable models and atlas-based approaches and classi-fiers (SVM KNN and decision trees) The presented evalua-tion framework provides an insight into the performance ofthese algorithms in terms of accuracy and robustness Variousfactors influence the choice for a certain method aboveothers We provide three measures that could aid in selectingthe method that is most appropriate for the segmentationgoal at hand a boundarymeasure (95th-percentile Hausdorffdistance 119867

95) an overlap measure (Dice coefficient 119863) and

a volume measure (absolute volume difference AVD) Allthree measures are taken into account for the final rankingof the methods This ranking was designed to get a quickinsight into how the methods perform in comparison to eachother The best overall method is the method that performs

well for all three measures and all three components (GMWM andCSF) However whichmethod to select depends onthe segmentation goal at hand Not all measures are relevantfor all segmentation goals For example if segmentation isused for brain volumetry [4] the overlap (119863) and volume(AVD) measures of the brain and intracranial volume (usedfor normalization [56]) segmentations are important to takeinto account On the other hand if segmentation is usedfor cortical thickness measurements the focus should be onthe gray matter boundary (119867

95) and overlap (119863) measures

Therefore the final ranking should be used to get a first insightinto the overall performance after which the performanceof the measures and components that are most relevant forthe segmentation goal at hand should be considered Besidesaccuracy robustness could also influence the choice for acertain method above others For example team UB VPMLMed shows a high sensitivity score for segmenting whitemat-ter lesions as white matter (Figure 6) and shows a consistentsegmentation performance of gray and white matter over all15 test datasets (Figures 2ndash4) This could be beneficial forsegmenting scans of populations with white matter lesionsbut is less important if the goal is to segment scans ofyoung healthy subjects In the latter case the most accuratesegmentation for gray and white matter (team BIGR2) ismore interesting If a segmentation algorithm is to be usedin clinical practice speed is an important consideration aswell The runtime of the evaluated methods is reported inTable 1 However these runtimes are merely an indicationof the required time since academic software is generallynot optimized for speed and the runtime is measured ondifferent computers andplatformsAnother relevant aspect ofthe evaluation framework is the comparison of multi- versussingle-sequence approaches For example most methodsstruggle with the segmentation of the intracranial volume onthe T1-weighted scan There is no contrast between the CSFand the skull and the contrast between the dura mater andthe CSF is not always sufficient Team Robarts used an atlas-based registration approach on the T1-IR scan (good contrastbetween skull and CSF) to segment the intracranial volumewhich resulted in the best performance for intracranialvolume segmentation (Table 1 Figures 2ndash4) Most methodsadd the T2-FLAIR scan to improve robustness against whitematter lesions (Table 1 Figure 6) Although using only theT1-weighted scan and incorporating prior shape information(team UofL BioImaging) can be very effective also thefreeware packages support this as well Since FreeSurfer isan atlas-based method it uses prior information and is themost robust of all freeware packages to white matter lesionsHowever adding the T2 FLAIR scan to SPM12 increasesrobustness against white matter lesions as well as comparedto applying SPM12 to the T1 scan only (Figure 6) In generalSPM12with the T1 and the T2-FLAIR sequence performswellin comparison to the other freeware packages (Table 1 andFigures 2ndash4) on the thick slice MRI scans Although addingthe T1-IR scan to SPM increases the performance of the CSFand ICV segmentations as compared to using only the T1 andT2-FLAIR sequence it decreases the performance of the GMand WM segmentations Therefore adding all sequences toSPM12 did not result in a better overall performance

Computational Intelligence and Neuroscience 7

Table 1 Results of the 11 evaluated algorithms presented at the workshop and the evaluated freeware packages on the 15 test datasets Thealgorithms are ranked (119903) based on their overall score (119904) by using (5) This score is based on the ranks of the gray matter (GM) white matter(WM) and cerebrospinal fluid (CSF) segmentation and the three evaluated measures Dice coefficient (119863 in ) 95th-percentile Hausdorffdistance (119867

95in mm) and the absolute volume difference (AVD in ) The rank 119903

119898119888denotes the rank based on the mean (120583) over all 15 test

datasets for each measure 119898 (0 119863 1 11986795 and 2 AVD) and component 119888 (0 GM 1 WM 2 CSF 3 brain (WM + GM) and 4 intracranial

volume (ICV = WM + GM + CSF)) Teams BIGR2 and UofL BioImaging and FreeSurfer and Jedi Mind Meld have equal scores based onthe mean (120583) therefore the ranking based on the standard deviation (120590) is taken into account to determine the final rank (BIGR2 120590 rank 4UofL BioImaging 120590 rank 8 FreeSurfer 120590 rank 13 and Jedi MindMeld 120590 rank 17) Columns 2 and 3 present the average runtime 119905 per scan inseconds (s) minutes (m) or hours (h) and the scans (T1 T1-weighted scan 3D T1 3D T1-weighted scan IR T1-weighted inversion recovery(IR) and F T1-weighted FLAIR) that are used for processing

119903 Team 119905 Scans

GM WM CSF

119904

Brain ICV119903001199031011990320119903011199031111990321119903021199031211990322

119903031199031311990323119903041199031411990324

119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590)

1 BIGR2 35m T1 IR F1 2 4 2 2 4 4 5 14

381 2 12 8 4 12

847 19 61 884 24 60 783 32 23 951 27 32 960 39 52(13) (04) (33) (12) (05) (51) (50) (08) (17) (05) (08) (16) (13) (11) (30)

2 UofLBioImaginglowast 6 s T1

5 1 9 4 1 13 2 2 138

2 1 12 5 2 5830 17 86 879 22 87 789 27 97 949 24 39 967 34 18(15) (03) (54) (20) (06) (66) (42) (05) (10) (06) (05) (20) (08) (06) (20)

3 CMIV 3m T1 F6 7 5 5 3 10 3 3 8

505 7 2 4 3 11

824 27 68 877 24 73 786 30 14 945 38 26 968 38 49(14) (04) (40) (16) (04) (38) (31) (04) (59) (05) (11) (22) (08) (13) (23)

4 UB VPMLMed 30m T1 IR F

4 4 2 1 5 7 8 13 1761

4 4 1 10 11 15833 21 59 886 27 71 748 43 31 946 28 24 948 66 77(13) (03) (53) (17) (04) (38) (71) (17) (19) (06) (04) (18) (20) (20) (41)

5 Bigr neuro 2 h T1 F7 13 3 6 6 9 6 4 10

647 10 10 7 5 8

815 37 59 873 30 73 782 32 16 940 46 36 963 39 35(17) (09) (42) (14) (04) (38) (47) (06) (14) (08) (14) (24) (12) (09) (27)

6 Robarts 16m 3D T1 IR11 3 15 8 8 6 1 1 13

6616 3 18 1 1 1

797 20 98 862 31 71 803 27 20 931 28 79 979 26 09(24) (01) (73) (13) (04) (62) (41) (05) (13) (16) (05) (36) (03) (04) (07)

7 Narsil 2m T1 F3 5 1 7 11 2 17 18 7

713 5 3 16 18 9

835 23 55 871 33 58 666 133 14 948 29 29 925 24 37(18) (04) (44) (13) (09) (53) (24) (54) (95) (05) (05) (20) (05) (89) (17)

8 SPM T1 F 3m T1 F8 9 16 9 7 1 9 14 2

759 14 14 6 14 4

812 29 10 860 30 52 741 46 10 939 58 53 966 82 15(22) (03) (85) (15) (01) (38) (34) (06) (47) (10) (22) (38) (02) (30) (10)

9 SPM T1 IR 3m T1 IR12 11 7 16 12 5 5 10 3

818 11 7 2 8 2

794 30 72 835 36 63 783 40 10 939 46 34 977 65 10(21) (04) (63) (21) (03) (46) (38) (06) (57) (08) (12) (28) (02) (13) (08)

10 MNAB 15m T1 IR F2 8 12 3 4 11 15 15 16

866 9 11 15 12 17

839 28 91 880 27 78 681 49 29 945 45 38 925 71 97(21) (09) (65) (12) (08) (40) (40) (22) (21) (10) (20) (32) (11) (42) (47)

11 SPM T1 3m T19 10 6 11 10 3 11 16 15

9110 8 6 9 13 13

803 30 69 856 31 60 707 53 23 939 44 32 953 81 55(24) (05) (68) (17) (01) (41) (38) (15) (157) (09) (16) (29) (09) (37) (37)

12 FSL Seg 10m T113 16 10 10 13 14 12 6 5

9913 13 4 12 6 6

787 43 86 860 37 115 699 34 12 933 55 30 942 53 34(22) (12) (63) (26) (08) (63) (28) (02) (103) (08) (14) (15) (08) (11) (15)

8 Computational Intelligence and Neuroscience

Table 1 Continued

119903 Team 119905 Scans

GM WM CSF

119904

Brain ICV119903001199031011990320119903011199031111990321119903021199031211990322

119903031199031311990323119903041199031411990324

119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590)

13 SPM T1 IR F 4m T1 IR F10 12 18 15 14 12 7 11 6

10512 15 13 3 15 3

801 30 139 836 38 84 769 41 12 936 59 51 977 82 12(24) (02) (96) (21) (05) (52) (31) (05) (60) (11) (18) (36) (02) (18) (09)

14 FSL PVSeg 10m T115 15 8 13 15 17 13 7 4

10711 16 9 13 7 7

777 43 84 848 38 197 695 34 11 936 61 35 942 53 34(26) (13) (65) (32) (09) (10) (22) (03) (58) (09) (16) (30) (08) (11) (15)

15 FreeSurfer 1 h 3D T116 6 17 12 9 8 18 12 18

11617 6 16 17 10 18

774 23 121 852 31 72 658 43 50 923 36 63 924 66 10(20) (06) (60) (22) (05) (46) (37) (06) (196) (08) (06) (34) (09) (04) (45)

16 Jedi MindMeld 27 s T1 IR F

14 14 11 17 16 15 10 8 11116

15 12 8 11 16 14778 39 89 806 45 137 733 37 18 932 54 35 943 20 68(59) (28) (76) (96) (36) (12) (35) (08) (90) (17) (48) (30) (17) (37) (31)

17 S2 QM 15 h T1 IR F17 17 13 14 17 18 16 9 9

13014 17 15 14 9 10

764 55 93 839 49 248 679 38 14 933 70 55 937 65 40(34) (30) (64) (34) (32) (10) (23) (06) (99) (14) (35) (45) (11) (14) (22)

18 LNMBrains 5m T1 IR F18 18 14 18 18 16 14 17 12

14518 18 17 18 17 16

728 68 95 783 68 153 688 77 20 885 86 74 892 204 79(53) (23) (76) (64) (35) (13) (66) (24) (13) (45) (28) (77) (48) (37) (83)

lowastSemiautomatic due to per scan parameter tuning

Besides the advantages of the MRBrainS evaluationframework there are some limitations that should be takeninto account The T1-weighted IR and the T2-weightedFLAIR scan were acquired with a lower resolution (096times 096 times 300mm3) than the 3D T1-weighted scan (10 times10 times 10mm3) To be able to provide a registered multise-quence dataset the 3D T1-weighted scan was registered tothe T2-weighted FLAIR scan and downsampled to 096 times096 times 300mm3 The reference standard is therefore onlyavailable for this resolution The decreased performance ofthe FreeSurfer GM segmentation as compared to the otherfreeware packages might be due to the fact that we evaluateon the thick slice T1 sequence instead of the high resolutionT1 Performing the manual segmentations to provide thereference standard is very laborsome and time consumingInstead of letting multiple observers manually segment theMRI datasets or letting one observer manually segmentthe MRI datasets twice much time and effort was spenton creating one reference standard that was as accurate aspossible Therefore we were not able to determine the inter-or intraobserver variability Finally we acknowledge that ourevaluation framework is limited to evaluating the accuracyand robustness over 15 datasets for segmenting GM WMand CSF on 3T MRI scans acquired on a Philips scanner ofa specific group of elderly subjects Many factors influencesegmentation algorithm performance such as the type ofscanner (vendor field strength) the acquisition protocolthe available MRI sequences and the type of subjects

Participating algorithmsmight have been designed for differ-ent types of MRI scans Therefore the five provided trainingdatasets are important for participants to be able to traintheir algorithms on the provided data Some algorithms aredesigned to segment only some components such as onlyGM and WM instead of all three components and usefreely available software such as the brain extraction tool[57] to segment the outer border of the CSF (intracranialvolume) We have chosen to base the final ranking on allthree components but it is therefore important to assess notonly the final ranking but the performance of the individualcomponents as well

Despite these limitations the MRBrainS evaluationframework provides an objective and direct comparison ofsegmentation algorithms The reference standard of the testdata is unknown to the participants the same evaluationmeasures are used for all evaluated algorithms and partici-pants apply their own algorithms to the provided data

In comparison to the online validation engine proposedby Shattuck et al [58] the MRBrainS evaluation frameworkuses 3T MRI data instead of 15T MRI data and evaluates notonly brain versus nonbrain segmentation but also segmenta-tion of gray matter white matter cerebrospinal fluid brainand intracranial volume The availability of many differenttypes of evaluation frameworks will aid in the developmentof more generic and robust algorithms For example inthe NEATBrainS (httpneatbrains15isiuunl) challengeresearchers were challenged to apply their algorithms to data

Computational Intelligence and Neuroscience 9

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

65

70

75

80

85

GM Dice

Dic

e (

)

(a)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

55

60

65

70

75

80

85

90

WM Dice

Dic

e (

)

(b)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

55

60

65

70

75

80

85CSF Dice

Dic

e (

)

(c)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

80

85

90

95

Brain (WM + GM) Dice

Dic

e (

)

(d)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

80

85

90

95

Dic

e (

)

Intracranial volume (WM + GM + CSF) Dice

(e)

Figure 2 Boxplots presenting the evaluation results for the Dice coefficient (1) of the gray matter (GM) white matter (WM) cerebrospinalfluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all 15 test datasets

10 Computational Intelligence and Neuroscience

2468

10121416

GM H-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(a)

2468

10121416

WMH-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(b)

5

10

15

20CSF H-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(c)

5

10

15

20H

-95

(mm

)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Brain (WM + GM) H-95

(d)

5

10

15

20

25

30

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Intracranial volume (WM + GM + CSF) H-95

(e)

Figure 3 Boxplots presenting the evaluation results for the 95th-percentile Hausdorff distance (3) of the gray matter (GM) white matter(WM) cerebrospinal fluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all15 test datasets

Computational Intelligence and Neuroscience 11

0

10

20

30

GM AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(a)

0

10

20

30

40WM AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(b)

0

20

40

60

80

CSF AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(c)

0

5

10

15

20

25

30AV

D (

)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Brain (WM + GM) AVD

(d)

0

5

10

15

20

25

30

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Intracranial volume (WM + GM + CSF) AVD

(e)

Figure 4 Boxplots presenting the evaluation results for the absolute volume difference (4) of the gray matter (GM) white matter (WM)cerebrospinal fluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all 15 testdatasets

12 Computational Intelligence and Neuroscience

T2

UofL

N

FS

S2

‐FLAIR

BioImaging

Narsil

L‐Seg

2_QM

T1‐IR

CMIV

SPM12_T1

SPM12_T1_

LNMBr

_FLAIR

_IR_FLAIR

ains

T1

UB VPML Me

SPM12_T1_ I

FSL‐PVSeg

R

d B

R

F

eference

igr_neuro

MNAB

reeSurfer

BIG

Rob

SPM1

Jedi Mi

R2

arts

2_T1

nd Meld

Figure 5 Illustration of the segmentation results at the height of the basal ganglia (test subject 9 slice 22) The basal ganglia should besegmented as gray matter The first three images show the three MRI sequences The fourth image (reference) is the manually segmentedreference standard (yellow white matter light blue gray matter and dark blue cerebrospinal fluid) The results of the participatingsegmentation algorithms are ordered from the best overall performance (BIGR2) to the worst overall performance (LNMBrains) Majordifferences are present in the segmentation of the basal ganglia and the outer border of the cerebrospinal fluid The arrows indicate examplelocations where the segmentation results differ from the ground truth

Computational Intelligence and Neuroscience 13

T2

UofL BioIm

Nars

FSL‐S

S2_Q

‐FLAIR

aging (860)

il (892)

eg (453)

M (565)

T1‐IR

CMIV (90

SPM12_T1_F

SPM12_T1_IR

LNMBrains

5) UB

(831) SP

_F (815) F

(286)

T1

VPML Med (9

M12_T1_IR (60

SL‐PVSeg (525

R

10) Bigr_

5) MN

) FreeS

eference

neuro (896)

AB (889)

urfer (846)

BIGR2

Robarts

SPM12_T

Jedi Mind M

(852)

(810)

1 (683)

eld (769)

Figure 6 Illustration of the segmentation results in the presence of whitematter lesions (test subject 3 slice 31)Whitematter lesions (WMLs)should be labeled asWM the sensitivity (percentage ofWML voxels is labeled asWM) over all 15 datasets is presented between brackets afterthe team nameThe first three images show the threeMRI sequences T2-weighted fluid attenuated inversion recovery T1-weighted inversionrecovery and T1-weighted scanThe fourth image (reference) is themanually segmented reference standard (red whitematter lesions yellowwhite matter light blue gray matter and dark blue cerebrospinal fluid)The results of the segmentation algorithms are ordered from the bestoverall performance (BIGR2) to the worst overall performance (LNMBrains)The arrows indicate example locations where the segmentationresults differ from the ground truth

14 Computational Intelligence and Neuroscience

fromboth theMRBrainS and theNeoBrainS [59] (brain tissuesegmentation in neonates) challenge Two methods [60ndash62]specifically designed for neonatal brain tissue segmentationshowed a high performance for tissue segmentation on theMRBrainS data Applying algorithms to different types of datahas the potential to lead to new insights and more robustalgorithms The MRBrainS evaluation framework remainsopen for new contributions At the time of writing 21teams had submitted their results on the MRBrainS website(httpmrbrains13isiuunlresultsphp)

5 Conclusion

The MRBrainS challenge online evaluation framework pro-vides direct and objective comparison of automatic andsemiautomatic methods to segment GM WM CSF brainand ICV on 3T multisequence MRI data The first elevenparticipating methods are evaluated and presented in thispaper as well as three commonly used freeware packages(FreeSurfer FSL and SPM12) They provide a benchmarkfor future contributions to the framework The final rankingprovides a quick insight into the overall performance of theevaluated methods in comparison to each other whereasthe individual evaluation measures (Dice 95th-percentileHausdorff distance and absolute volume difference) percomponent (GM WM CSF brain and ICV) can aid inselecting the best method for a specific segmentation goal

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was financially supported by IMDI Grant104002002 (Brainbox) fromZonMw the Netherlands Organ-isation for Health Research and Development within kindsponsoring by Philips theUniversityMedical CenterUtrechtand Eindhoven University of TechnologyThe authors wouldlike to acknowledge the following members of the UtrechtVascular Cognitive Impairment Study Group who were notincluded as coauthors of this paper but were involved in therecruitment of study participants and MRI acquisition at theUMC Utrecht (in alphabetical order by department) E vanden Berg M Brundel S Heringa and L J Kappelle of theDepartment of Neurology P R Luijten and W P Th MMali of the Department of Radiology and A Algra and GE H M Rutten of the Julius Center for Health Sciences andPrimary CareThe research of Geert Jan Biessels and the VCIgroupwas financially supported byVIDIGrant 91711384 fromZonMw and by Grant 2010T073 of the Netherlands HeartFoundation The research of Jeroen de Bresser is financiallysupported by a research talent fellowship of the Univer-sity Medical Center Utrecht (Netherlands) The research ofAnnegreet vanOpbroek andMarleen de Bruijne is financiallysupported by a research grant from NWO (the NetherlandsOrganisation for Scientific Research) The authors would

like to acknowledge MeVis Medical Solutions AG (BremenGermany) for providing MeVisLab Duygu Sarikaya andLiang Zhao acknowledge their Advisor Professor JasonCorsofor his guidance Duygu Sarikaya is supported by NIH 1 R21CA160825-01 and Liang Zhao is partially supported by theChina Scholarship Council (CSC)

References

[1] I Driscoll C Davatzikos Y An et al ldquoLongitudinal pattern ofregional brain volume change differentiates normal aging fromMCIrdquo Neurology vol 72 no 22 pp 1906ndash1913 2009

[2] K Franke and C Gaser ldquoLongitudinal changes in individualBrainAGE in healthy aging mild cognitive impairment andAlzheimerrsquos diseaserdquoGeroPsych vol 25 no 4 pp 235ndash245 2012

[3] MA IkramH A VroomanMWVernooij et al ldquoBrain tissuevolumes in the general elderly populationThe Rotterdam ScanStudyrdquo Neurobiology of Aging vol 29 no 6 pp 882ndash890 2008

[4] A Giorgio and N de Stefano ldquoClinical use of brain volumetryrdquoJournal of Magnetic Resonance Imaging vol 37 no 1 pp 1ndash142013

[5] M W Vannier R L Butterfield D Jordan W A Murphy RG Levitt and M Gado ldquoMultispectral analysis of magneticresonance imagesrdquo Radiology vol 154 no 1 pp 221ndash224 1985

[6] M A Balafar A R Ramli M I Saripan and S MashohorldquoReview of brain MRI image segmentation methodsrdquo ArtificialIntelligence Review vol 33 no 3 pp 261ndash274 2010

[7] J C Bezdek L O Hall and L P Clarke ldquoReview of MR imagesegmentation techniques using pattern recognitionrdquo MedicalPhysics vol 20 no 4 pp 1033ndash1048 1993

[8] L P Clarke R P Velthuizen M A Camacho et al ldquoMRIsegmentation methods and applicationsrdquo Magnetic ResonanceImaging vol 13 no 3 pp 343ndash368 1995

[9] J S Duncan X Papademetris J Yang M Jackowski X Zengand L H Staib ldquoGeometric strategies for neuroanatomic ana-lysis from MRIrdquo NeuroImage vol 23 supplement 1 pp S34ndashS45 2004

[10] D L PhamC Xu and J L Prince ldquoCurrentmethods inmedicalimage segmentationrdquoAnnual Review of Biomedical Engineeringvol 2 no 2000 pp 315ndash337 2000

[11] N Saeed ldquoMagnetic resonance image segmentation usingpattern recognition and applied to image registration andquantitationrdquo NMR in Biomedicine vol 11 no 4-5 pp 157ndash1671998

[12] J S Suri S Singh and L Reden ldquoComputer vision andpattern recognition techniques for 2-D and 3-D MR cerebralcortical segmentation (Part I) a state-of-the-art reviewrdquo PatternAnalysis amp Applications vol 5 no 1 pp 46ndash76 2002

[13] A P Zijdenbos and B M Dawant ldquoBrain segmentation andwhite matter lesion detection in MR imagesrdquo Critical Reviewsin Biomedical Engineering vol 22 no 5-6 pp 401ndash465 1994

[14] K Price ldquoAnything you can do I can do better (No you canrsquot)rdquoComputer Vision Graphics and Image Processing vol 36 no 2-3pp 387ndash391 1986

[15] R de Boer H A Vrooman M A Ikram et al ldquoAccuracy andreproducibility study of automatic MRI brain tissue segmenta-tion methodsrdquo NeuroImage vol 51 no 3 pp 1047ndash1056 2010

[16] J de Bresser M P Portegies A Leemans G J Biessels L JKappelle andM A Viergever ldquoA comparison ofMR based seg-mentation methods for measuring brain atrophy progressionrdquoNeuroImage vol 54 no 2 pp 760ndash768 2011

Computational Intelligence and Neuroscience 15

[17] M B Cuadra L Cammoun T Butz O Cuisenaire and J-P Thiran ldquoComparison and validation of tissue modelizationand statistical classification methods in T1-weighted MR brainimagesrdquo IEEE Transactions on Medical Imaging vol 24 no 12pp 1548ndash1565 2005

[18] F Klauschen A Goldman V Barra A Meyer-Lindenbergand A Lundervold ldquoEvaluation of automated brain MR imagesegmentation and volumetrymethodsrdquoHuman BrainMappingvol 30 no 4 pp 1310ndash1327 2009

[19] H Zaidi T Ruest F Schoenahl and M-L Montandon ldquoCom-parative assessment of statistical brain MR image segmentationalgorithms and their impact on partial volume correction inPETrdquo NeuroImage vol 32 no 4 pp 1591ndash1607 2006

[20] J A Frazier S M Hodge J L Breeze et al ldquoDiagnostic and sexeffects on limbic volumes in early-onset bipolar disorder andschizophreniardquo Schizophrenia Bulletin vol 34 no 1 pp 37ndash462008

[21] A Klein and J Tourville ldquo101 labeled brain images and aconsistent human cortical labeling protocolrdquo Frontiers in Neu-roscience vol 6 article 171 2012

[22] B Van Ginneken T Heimann and M Styner ldquo3D segmen-tation in the clinic a grand challengerdquo in Proceedings of the10th International Conference on Medical Image Computingand Computer Assisted Intervention (MICCAI rsquo07) pp 7ndash15Brisbane Australia October-November 2007

[23] A M Dale B Fischl and M I Sereno ldquoCortical surface-based analysis I Segmentation and surface reconstructionrdquoNeuroImage vol 9 no 2 pp 179ndash194 1999

[24] B Fischl M I Sereno and A M Dale ldquoCortical surface-basedanalysis II inflation flattening and a surface-based coordinatesystemrdquo NeuroImage vol 9 no 2 pp 195ndash207 1999

[25] M Jenkinson C F Beckmann T E J BehrensMWWoolrichand S M Smith ldquoFSLrdquoNeuroImage vol 62 no 2 pp 782ndash7902012

[26] J Ashburner and K J Friston ldquoUnified segmentationrdquo Neu-roImage vol 26 no 3 pp 839ndash851 2005

[27] Y D Reijmer A Leemans M Brundel L J Kappelle and GJ Biessels ldquoDisruption of the cerebral white matter network isrelated to slowing of information processing speed in patientswith type 2 diabetesrdquoDiabetes vol 62 no 6 pp 2112ndash2115 2013

[28] S Klein M Staring K Murphy M A Viergever and J P WPluim ldquoElastix a toolbox for intensity-based medical imageregistrationrdquo IEEE Transactions onMedical Imaging vol 29 no1 pp 196ndash205 2010

[29] K J Friston J T Ashburner S J Kiebel T E Nichols and WD Penny Eds Statistical Parametric Mapping The Analysis ofFunctional Brain Images Academic Press 2011

[30] F Ritter T Boskamp A Homeyer et al ldquoMedical imageanalysis a visual approachrdquo IEEE Pulse vol 2 no 6 pp 60ndash702011

[31] L R Dice ldquoMeasures of the amount of ecologic associationbetween speciesrdquo Ecology vol 26 no 3 pp 297ndash302 1945

[32] D P Huttenlocher G A Klanderman and W J RucklidgeldquoComparing images using the Hausdorff distancerdquo IEEE Trans-actions on Pattern Analysis and Machine Intelligence vol 15 no9 pp 850ndash863 1993

[33] A Alansary A Soliman F Khalifa et al ldquoMAP-based frame-work for segmentation of MR brain images based on visualappearance and prior shaperdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal September 2013

[34] A Jog S Roy J L Prince and A Carass ldquoMR brain segmenta-tion using decision treesrdquo in Proceedings of the MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal Nagoya Japan September 2013

[35] R Katyal S Paneri and M Kuse ldquoGaussian intensity modelwith neighborhood cues for fluid-tissue categorization ofmulti-sequence MR brain imagesrdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal Nagoya Japan September 2013

[36] QMahmoodMAlipoor A Chodorowski AMehnert andMPersson ldquoMultimodal MR brain segmentation using Bayesian-based adaptive mean-shift (BAMS)rdquo in Proceedings of theMICCAI WorkshopsmdashThe MICCAI Grand Challenge on MRBrain Image Segmentation (MRBrainS rsquo13) The Midas JournalSeptember 2013

[37] A van Opbroek F van der Lijn andM de Bruijne ldquoAutomatedbrain-tissue segmentation by multi-feature SVM classificationrdquoin Proceedings of the MICCAI WorkshopsmdashThe MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal September 2013

[38] S Pereira J Festa J A Mariz N Sousa and C A SilvaldquoAutomatic brain tissue segmentation of multi-sequence MRimages using random decision forestsrdquo in Proceedings of theMICCAI Grand Challenge on MR Brain Image Segmentation(MRBrainS rsquo13) The Midas Journal Nagoya Japan September2013

[39] MRajchl J Baxter J Yuan T Peters andAKhan ldquoMulti-atlas-based segmentation with hierarchical max-flowrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[40] D Sarikaya L Zhao and J J Corso ldquoMulti-atlas brain MRIsegmentation with multiway cutrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[41] H Vrooman F van der Lijn and W Niessen ldquoAuto-kNNbrain tissue segmentation using automatically trained k-nearest-neighbor classificationrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[42] S Vyas P Burlina D Kleissas and R Mukherjee ldquoAutomatedwalks using machine learning for segmentationrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[43] C Wang and O Smedby ldquoFully automatic brain segmentationusing model-guided level sets and skeleton-based modelsrdquo inProceedings of theMICCAI Grand Challenge onMRBrain ImageSegmentation (MRBrainS rsquo13) The Midas Journal NagoyaJapan September 2013

[44] C Cortes and V Vapnik ldquoSupport-vector networksrdquo MachineLearning vol 20 no 3 pp 273ndash297 1995

[45] H A Vrooman C A Cocosco F van der Lijn et al ldquoMulti-spectral brain tissue segmentation using automatically trainedk-Nearest-Neighbor classificationrdquo NeuroImage vol 37 no 1pp 71ndash81 2007

[46] C Wang H Frimmel and O Smedby ldquoFast level-set basedimage segmentation using coherent propagationrdquo MedicalPhysics vol 41 no 7 Article ID 073501 2014

16 Computational Intelligence and Neuroscience

[47] A Jog S Roy A Carass and J L Prince ldquoMagnetic resonanceimage synthesis through patch regressionrdquo in Proceedings of theIEEE 10th International Symposium onBiomedical Imaging (ISBIrsquo13) pp 350ndash353 April 2013

[48] J G Sied A P Zijdenbos and A C Evans ldquoA non-parametricmethod for automatic correction of intensity non-uniformity inMRI datardquo IEEE Transactions on Medical Imaging vol 17 no 1pp 87ndash97 1998

[49] A Carass J Cuzzocreo M B Wheeler P-L Bazin S MResnick and J L Prince ldquoSimple paradigm for extra-cerebraltissue removal algorithm and analysisrdquoNeuroImage vol 56 no4 pp 1982ndash1992 2011

[50] M Modat G R Ridgway Z A Taylor et al ldquoFast free-form deformation using graphics processing unitsrdquo ComputerMethods and Programs in Biomedicine vol 98 no 3 pp 278ndash284 2010

[51] M Rajchl J Yuan J A White et al ldquoInteractive hierarchical-flow segmentation of scar tissue from late-enhancement cardiacmr imagesrdquo IEEE Transactions on Medical Imaging vol 33 no1 pp 159ndash172 2014

[52] QMahmood A Chodorowski AMehnert andM Persson ldquoAnovel Bayesian approach to adaptive mean shift segmentationof brain imagesrdquo in Proceedings of the 25th IEEE InternationalSymposiumonComputer-BasedMedical Systems (CBMS rsquo12) pp1ndash6 June 2012

[53] A El-Baz M Nitzken A Alansary A Soliman and MCasanova ldquoBrain Segmentation Method for Young Childrenand Adultsrdquo US Provisional Patent Application 13100

[54] A Alansary A Soliman M Nitzken et al ldquoAn integratedgeometrical and stochastic approach for accurate infant brainextractionrdquo in Proceedings of the IEEE International Conferenceon Image Processing (ICIP rsquo14) pp 3542ndash3546 October 2014

[55] V Popescu M Battaglini W S Hoogstrate et al ldquoOptimizingparameter choice for FSL-Brain Extraction Tool (BET) on 3DT1 images in multiple sclerosisrdquo NeuroImage vol 61 no 4 pp1484ndash1494 2012

[56] J L Whitwell W R Crum H C Watt and N C FoxldquoNormalization of cerebral volumes by use of intracranial vol-ume implications for longitudinal quantitative MR imagingrdquoAmerican Journal of Neuroradiology vol 22 no 8 pp 1483ndash1489 2001

[57] S M Smith ldquoFast robust automated brain extractionrdquo HumanBrain Mapping vol 17 no 3 pp 143ndash155 2002

[58] D W Shattuck G Prasad M Mirza K L Narr and A WToga ldquoOnline resource for validation of brain segmentationmethodsrdquo NeuroImage vol 45 no 2 pp 431ndash439 2009

[59] I Isgum M J N L Benders B Avants et al ldquoEvaluation ofautomatic neonatal brain segmentation algorithms the Neo-BrainS12 challengerdquo Medical Image Analysis vol 20 no 1 pp135ndash151 2015

[60] P Moeskops M J N L Benders S M Chita et al ldquoAutomaticsegmentation of MR brain images of preterm infants usingsupervised classificationrdquo NeuroImage 2015

[61] P Moeskops M A Viergever M J N L Benders and IIsgum ldquoEvaluation of an automatic brain segmentationmethoddeveloped for neonates on adult MR brain imagesrdquo in MedicalImaging Image Processing vol 9413 of Proceedings of SPIEInternational Society for Optics and Photonics Orlando FlaUSA February 2015

[62] L Wang Y Gao F Shi et al ldquoLINKS learning-based multi-source IntegratioN frameworK for Segmentation of infant brainimagesrdquo NeuroImage vol 108 pp 160ndash172 2015

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience 7

Table 1 Results of the 11 evaluated algorithms presented at the workshop and the evaluated freeware packages on the 15 test datasets Thealgorithms are ranked (119903) based on their overall score (119904) by using (5) This score is based on the ranks of the gray matter (GM) white matter(WM) and cerebrospinal fluid (CSF) segmentation and the three evaluated measures Dice coefficient (119863 in ) 95th-percentile Hausdorffdistance (119867

95in mm) and the absolute volume difference (AVD in ) The rank 119903

119898119888denotes the rank based on the mean (120583) over all 15 test

datasets for each measure 119898 (0 119863 1 11986795 and 2 AVD) and component 119888 (0 GM 1 WM 2 CSF 3 brain (WM + GM) and 4 intracranial

volume (ICV = WM + GM + CSF)) Teams BIGR2 and UofL BioImaging and FreeSurfer and Jedi Mind Meld have equal scores based onthe mean (120583) therefore the ranking based on the standard deviation (120590) is taken into account to determine the final rank (BIGR2 120590 rank 4UofL BioImaging 120590 rank 8 FreeSurfer 120590 rank 13 and Jedi MindMeld 120590 rank 17) Columns 2 and 3 present the average runtime 119905 per scan inseconds (s) minutes (m) or hours (h) and the scans (T1 T1-weighted scan 3D T1 3D T1-weighted scan IR T1-weighted inversion recovery(IR) and F T1-weighted FLAIR) that are used for processing

119903 Team 119905 Scans

GM WM CSF

119904

Brain ICV119903001199031011990320119903011199031111990321119903021199031211990322

119903031199031311990323119903041199031411990324

119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590)

1 BIGR2 35m T1 IR F1 2 4 2 2 4 4 5 14

381 2 12 8 4 12

847 19 61 884 24 60 783 32 23 951 27 32 960 39 52(13) (04) (33) (12) (05) (51) (50) (08) (17) (05) (08) (16) (13) (11) (30)

2 UofLBioImaginglowast 6 s T1

5 1 9 4 1 13 2 2 138

2 1 12 5 2 5830 17 86 879 22 87 789 27 97 949 24 39 967 34 18(15) (03) (54) (20) (06) (66) (42) (05) (10) (06) (05) (20) (08) (06) (20)

3 CMIV 3m T1 F6 7 5 5 3 10 3 3 8

505 7 2 4 3 11

824 27 68 877 24 73 786 30 14 945 38 26 968 38 49(14) (04) (40) (16) (04) (38) (31) (04) (59) (05) (11) (22) (08) (13) (23)

4 UB VPMLMed 30m T1 IR F

4 4 2 1 5 7 8 13 1761

4 4 1 10 11 15833 21 59 886 27 71 748 43 31 946 28 24 948 66 77(13) (03) (53) (17) (04) (38) (71) (17) (19) (06) (04) (18) (20) (20) (41)

5 Bigr neuro 2 h T1 F7 13 3 6 6 9 6 4 10

647 10 10 7 5 8

815 37 59 873 30 73 782 32 16 940 46 36 963 39 35(17) (09) (42) (14) (04) (38) (47) (06) (14) (08) (14) (24) (12) (09) (27)

6 Robarts 16m 3D T1 IR11 3 15 8 8 6 1 1 13

6616 3 18 1 1 1

797 20 98 862 31 71 803 27 20 931 28 79 979 26 09(24) (01) (73) (13) (04) (62) (41) (05) (13) (16) (05) (36) (03) (04) (07)

7 Narsil 2m T1 F3 5 1 7 11 2 17 18 7

713 5 3 16 18 9

835 23 55 871 33 58 666 133 14 948 29 29 925 24 37(18) (04) (44) (13) (09) (53) (24) (54) (95) (05) (05) (20) (05) (89) (17)

8 SPM T1 F 3m T1 F8 9 16 9 7 1 9 14 2

759 14 14 6 14 4

812 29 10 860 30 52 741 46 10 939 58 53 966 82 15(22) (03) (85) (15) (01) (38) (34) (06) (47) (10) (22) (38) (02) (30) (10)

9 SPM T1 IR 3m T1 IR12 11 7 16 12 5 5 10 3

818 11 7 2 8 2

794 30 72 835 36 63 783 40 10 939 46 34 977 65 10(21) (04) (63) (21) (03) (46) (38) (06) (57) (08) (12) (28) (02) (13) (08)

10 MNAB 15m T1 IR F2 8 12 3 4 11 15 15 16

866 9 11 15 12 17

839 28 91 880 27 78 681 49 29 945 45 38 925 71 97(21) (09) (65) (12) (08) (40) (40) (22) (21) (10) (20) (32) (11) (42) (47)

11 SPM T1 3m T19 10 6 11 10 3 11 16 15

9110 8 6 9 13 13

803 30 69 856 31 60 707 53 23 939 44 32 953 81 55(24) (05) (68) (17) (01) (41) (38) (15) (157) (09) (16) (29) (09) (37) (37)

12 FSL Seg 10m T113 16 10 10 13 14 12 6 5

9913 13 4 12 6 6

787 43 86 860 37 115 699 34 12 933 55 30 942 53 34(22) (12) (63) (26) (08) (63) (28) (02) (103) (08) (14) (15) (08) (11) (15)

8 Computational Intelligence and Neuroscience

Table 1 Continued

119903 Team 119905 Scans

GM WM CSF

119904

Brain ICV119903001199031011990320119903011199031111990321119903021199031211990322

119903031199031311990323119903041199031411990324

119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590)

13 SPM T1 IR F 4m T1 IR F10 12 18 15 14 12 7 11 6

10512 15 13 3 15 3

801 30 139 836 38 84 769 41 12 936 59 51 977 82 12(24) (02) (96) (21) (05) (52) (31) (05) (60) (11) (18) (36) (02) (18) (09)

14 FSL PVSeg 10m T115 15 8 13 15 17 13 7 4

10711 16 9 13 7 7

777 43 84 848 38 197 695 34 11 936 61 35 942 53 34(26) (13) (65) (32) (09) (10) (22) (03) (58) (09) (16) (30) (08) (11) (15)

15 FreeSurfer 1 h 3D T116 6 17 12 9 8 18 12 18

11617 6 16 17 10 18

774 23 121 852 31 72 658 43 50 923 36 63 924 66 10(20) (06) (60) (22) (05) (46) (37) (06) (196) (08) (06) (34) (09) (04) (45)

16 Jedi MindMeld 27 s T1 IR F

14 14 11 17 16 15 10 8 11116

15 12 8 11 16 14778 39 89 806 45 137 733 37 18 932 54 35 943 20 68(59) (28) (76) (96) (36) (12) (35) (08) (90) (17) (48) (30) (17) (37) (31)

17 S2 QM 15 h T1 IR F17 17 13 14 17 18 16 9 9

13014 17 15 14 9 10

764 55 93 839 49 248 679 38 14 933 70 55 937 65 40(34) (30) (64) (34) (32) (10) (23) (06) (99) (14) (35) (45) (11) (14) (22)

18 LNMBrains 5m T1 IR F18 18 14 18 18 16 14 17 12

14518 18 17 18 17 16

728 68 95 783 68 153 688 77 20 885 86 74 892 204 79(53) (23) (76) (64) (35) (13) (66) (24) (13) (45) (28) (77) (48) (37) (83)

lowastSemiautomatic due to per scan parameter tuning

Besides the advantages of the MRBrainS evaluationframework there are some limitations that should be takeninto account The T1-weighted IR and the T2-weightedFLAIR scan were acquired with a lower resolution (096times 096 times 300mm3) than the 3D T1-weighted scan (10 times10 times 10mm3) To be able to provide a registered multise-quence dataset the 3D T1-weighted scan was registered tothe T2-weighted FLAIR scan and downsampled to 096 times096 times 300mm3 The reference standard is therefore onlyavailable for this resolution The decreased performance ofthe FreeSurfer GM segmentation as compared to the otherfreeware packages might be due to the fact that we evaluateon the thick slice T1 sequence instead of the high resolutionT1 Performing the manual segmentations to provide thereference standard is very laborsome and time consumingInstead of letting multiple observers manually segment theMRI datasets or letting one observer manually segmentthe MRI datasets twice much time and effort was spenton creating one reference standard that was as accurate aspossible Therefore we were not able to determine the inter-or intraobserver variability Finally we acknowledge that ourevaluation framework is limited to evaluating the accuracyand robustness over 15 datasets for segmenting GM WMand CSF on 3T MRI scans acquired on a Philips scanner ofa specific group of elderly subjects Many factors influencesegmentation algorithm performance such as the type ofscanner (vendor field strength) the acquisition protocolthe available MRI sequences and the type of subjects

Participating algorithmsmight have been designed for differ-ent types of MRI scans Therefore the five provided trainingdatasets are important for participants to be able to traintheir algorithms on the provided data Some algorithms aredesigned to segment only some components such as onlyGM and WM instead of all three components and usefreely available software such as the brain extraction tool[57] to segment the outer border of the CSF (intracranialvolume) We have chosen to base the final ranking on allthree components but it is therefore important to assess notonly the final ranking but the performance of the individualcomponents as well

Despite these limitations the MRBrainS evaluationframework provides an objective and direct comparison ofsegmentation algorithms The reference standard of the testdata is unknown to the participants the same evaluationmeasures are used for all evaluated algorithms and partici-pants apply their own algorithms to the provided data

In comparison to the online validation engine proposedby Shattuck et al [58] the MRBrainS evaluation frameworkuses 3T MRI data instead of 15T MRI data and evaluates notonly brain versus nonbrain segmentation but also segmenta-tion of gray matter white matter cerebrospinal fluid brainand intracranial volume The availability of many differenttypes of evaluation frameworks will aid in the developmentof more generic and robust algorithms For example inthe NEATBrainS (httpneatbrains15isiuunl) challengeresearchers were challenged to apply their algorithms to data

Computational Intelligence and Neuroscience 9

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

65

70

75

80

85

GM Dice

Dic

e (

)

(a)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

55

60

65

70

75

80

85

90

WM Dice

Dic

e (

)

(b)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

55

60

65

70

75

80

85CSF Dice

Dic

e (

)

(c)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

80

85

90

95

Brain (WM + GM) Dice

Dic

e (

)

(d)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

80

85

90

95

Dic

e (

)

Intracranial volume (WM + GM + CSF) Dice

(e)

Figure 2 Boxplots presenting the evaluation results for the Dice coefficient (1) of the gray matter (GM) white matter (WM) cerebrospinalfluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all 15 test datasets

10 Computational Intelligence and Neuroscience

2468

10121416

GM H-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(a)

2468

10121416

WMH-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(b)

5

10

15

20CSF H-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(c)

5

10

15

20H

-95

(mm

)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Brain (WM + GM) H-95

(d)

5

10

15

20

25

30

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Intracranial volume (WM + GM + CSF) H-95

(e)

Figure 3 Boxplots presenting the evaluation results for the 95th-percentile Hausdorff distance (3) of the gray matter (GM) white matter(WM) cerebrospinal fluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all15 test datasets

Computational Intelligence and Neuroscience 11

0

10

20

30

GM AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(a)

0

10

20

30

40WM AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(b)

0

20

40

60

80

CSF AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(c)

0

5

10

15

20

25

30AV

D (

)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Brain (WM + GM) AVD

(d)

0

5

10

15

20

25

30

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Intracranial volume (WM + GM + CSF) AVD

(e)

Figure 4 Boxplots presenting the evaluation results for the absolute volume difference (4) of the gray matter (GM) white matter (WM)cerebrospinal fluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all 15 testdatasets

12 Computational Intelligence and Neuroscience

T2

UofL

N

FS

S2

‐FLAIR

BioImaging

Narsil

L‐Seg

2_QM

T1‐IR

CMIV

SPM12_T1

SPM12_T1_

LNMBr

_FLAIR

_IR_FLAIR

ains

T1

UB VPML Me

SPM12_T1_ I

FSL‐PVSeg

R

d B

R

F

eference

igr_neuro

MNAB

reeSurfer

BIG

Rob

SPM1

Jedi Mi

R2

arts

2_T1

nd Meld

Figure 5 Illustration of the segmentation results at the height of the basal ganglia (test subject 9 slice 22) The basal ganglia should besegmented as gray matter The first three images show the three MRI sequences The fourth image (reference) is the manually segmentedreference standard (yellow white matter light blue gray matter and dark blue cerebrospinal fluid) The results of the participatingsegmentation algorithms are ordered from the best overall performance (BIGR2) to the worst overall performance (LNMBrains) Majordifferences are present in the segmentation of the basal ganglia and the outer border of the cerebrospinal fluid The arrows indicate examplelocations where the segmentation results differ from the ground truth

Computational Intelligence and Neuroscience 13

T2

UofL BioIm

Nars

FSL‐S

S2_Q

‐FLAIR

aging (860)

il (892)

eg (453)

M (565)

T1‐IR

CMIV (90

SPM12_T1_F

SPM12_T1_IR

LNMBrains

5) UB

(831) SP

_F (815) F

(286)

T1

VPML Med (9

M12_T1_IR (60

SL‐PVSeg (525

R

10) Bigr_

5) MN

) FreeS

eference

neuro (896)

AB (889)

urfer (846)

BIGR2

Robarts

SPM12_T

Jedi Mind M

(852)

(810)

1 (683)

eld (769)

Figure 6 Illustration of the segmentation results in the presence of whitematter lesions (test subject 3 slice 31)Whitematter lesions (WMLs)should be labeled asWM the sensitivity (percentage ofWML voxels is labeled asWM) over all 15 datasets is presented between brackets afterthe team nameThe first three images show the threeMRI sequences T2-weighted fluid attenuated inversion recovery T1-weighted inversionrecovery and T1-weighted scanThe fourth image (reference) is themanually segmented reference standard (red whitematter lesions yellowwhite matter light blue gray matter and dark blue cerebrospinal fluid)The results of the segmentation algorithms are ordered from the bestoverall performance (BIGR2) to the worst overall performance (LNMBrains)The arrows indicate example locations where the segmentationresults differ from the ground truth

14 Computational Intelligence and Neuroscience

fromboth theMRBrainS and theNeoBrainS [59] (brain tissuesegmentation in neonates) challenge Two methods [60ndash62]specifically designed for neonatal brain tissue segmentationshowed a high performance for tissue segmentation on theMRBrainS data Applying algorithms to different types of datahas the potential to lead to new insights and more robustalgorithms The MRBrainS evaluation framework remainsopen for new contributions At the time of writing 21teams had submitted their results on the MRBrainS website(httpmrbrains13isiuunlresultsphp)

5 Conclusion

The MRBrainS challenge online evaluation framework pro-vides direct and objective comparison of automatic andsemiautomatic methods to segment GM WM CSF brainand ICV on 3T multisequence MRI data The first elevenparticipating methods are evaluated and presented in thispaper as well as three commonly used freeware packages(FreeSurfer FSL and SPM12) They provide a benchmarkfor future contributions to the framework The final rankingprovides a quick insight into the overall performance of theevaluated methods in comparison to each other whereasthe individual evaluation measures (Dice 95th-percentileHausdorff distance and absolute volume difference) percomponent (GM WM CSF brain and ICV) can aid inselecting the best method for a specific segmentation goal

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was financially supported by IMDI Grant104002002 (Brainbox) fromZonMw the Netherlands Organ-isation for Health Research and Development within kindsponsoring by Philips theUniversityMedical CenterUtrechtand Eindhoven University of TechnologyThe authors wouldlike to acknowledge the following members of the UtrechtVascular Cognitive Impairment Study Group who were notincluded as coauthors of this paper but were involved in therecruitment of study participants and MRI acquisition at theUMC Utrecht (in alphabetical order by department) E vanden Berg M Brundel S Heringa and L J Kappelle of theDepartment of Neurology P R Luijten and W P Th MMali of the Department of Radiology and A Algra and GE H M Rutten of the Julius Center for Health Sciences andPrimary CareThe research of Geert Jan Biessels and the VCIgroupwas financially supported byVIDIGrant 91711384 fromZonMw and by Grant 2010T073 of the Netherlands HeartFoundation The research of Jeroen de Bresser is financiallysupported by a research talent fellowship of the Univer-sity Medical Center Utrecht (Netherlands) The research ofAnnegreet vanOpbroek andMarleen de Bruijne is financiallysupported by a research grant from NWO (the NetherlandsOrganisation for Scientific Research) The authors would

like to acknowledge MeVis Medical Solutions AG (BremenGermany) for providing MeVisLab Duygu Sarikaya andLiang Zhao acknowledge their Advisor Professor JasonCorsofor his guidance Duygu Sarikaya is supported by NIH 1 R21CA160825-01 and Liang Zhao is partially supported by theChina Scholarship Council (CSC)

References

[1] I Driscoll C Davatzikos Y An et al ldquoLongitudinal pattern ofregional brain volume change differentiates normal aging fromMCIrdquo Neurology vol 72 no 22 pp 1906ndash1913 2009

[2] K Franke and C Gaser ldquoLongitudinal changes in individualBrainAGE in healthy aging mild cognitive impairment andAlzheimerrsquos diseaserdquoGeroPsych vol 25 no 4 pp 235ndash245 2012

[3] MA IkramH A VroomanMWVernooij et al ldquoBrain tissuevolumes in the general elderly populationThe Rotterdam ScanStudyrdquo Neurobiology of Aging vol 29 no 6 pp 882ndash890 2008

[4] A Giorgio and N de Stefano ldquoClinical use of brain volumetryrdquoJournal of Magnetic Resonance Imaging vol 37 no 1 pp 1ndash142013

[5] M W Vannier R L Butterfield D Jordan W A Murphy RG Levitt and M Gado ldquoMultispectral analysis of magneticresonance imagesrdquo Radiology vol 154 no 1 pp 221ndash224 1985

[6] M A Balafar A R Ramli M I Saripan and S MashohorldquoReview of brain MRI image segmentation methodsrdquo ArtificialIntelligence Review vol 33 no 3 pp 261ndash274 2010

[7] J C Bezdek L O Hall and L P Clarke ldquoReview of MR imagesegmentation techniques using pattern recognitionrdquo MedicalPhysics vol 20 no 4 pp 1033ndash1048 1993

[8] L P Clarke R P Velthuizen M A Camacho et al ldquoMRIsegmentation methods and applicationsrdquo Magnetic ResonanceImaging vol 13 no 3 pp 343ndash368 1995

[9] J S Duncan X Papademetris J Yang M Jackowski X Zengand L H Staib ldquoGeometric strategies for neuroanatomic ana-lysis from MRIrdquo NeuroImage vol 23 supplement 1 pp S34ndashS45 2004

[10] D L PhamC Xu and J L Prince ldquoCurrentmethods inmedicalimage segmentationrdquoAnnual Review of Biomedical Engineeringvol 2 no 2000 pp 315ndash337 2000

[11] N Saeed ldquoMagnetic resonance image segmentation usingpattern recognition and applied to image registration andquantitationrdquo NMR in Biomedicine vol 11 no 4-5 pp 157ndash1671998

[12] J S Suri S Singh and L Reden ldquoComputer vision andpattern recognition techniques for 2-D and 3-D MR cerebralcortical segmentation (Part I) a state-of-the-art reviewrdquo PatternAnalysis amp Applications vol 5 no 1 pp 46ndash76 2002

[13] A P Zijdenbos and B M Dawant ldquoBrain segmentation andwhite matter lesion detection in MR imagesrdquo Critical Reviewsin Biomedical Engineering vol 22 no 5-6 pp 401ndash465 1994

[14] K Price ldquoAnything you can do I can do better (No you canrsquot)rdquoComputer Vision Graphics and Image Processing vol 36 no 2-3pp 387ndash391 1986

[15] R de Boer H A Vrooman M A Ikram et al ldquoAccuracy andreproducibility study of automatic MRI brain tissue segmenta-tion methodsrdquo NeuroImage vol 51 no 3 pp 1047ndash1056 2010

[16] J de Bresser M P Portegies A Leemans G J Biessels L JKappelle andM A Viergever ldquoA comparison ofMR based seg-mentation methods for measuring brain atrophy progressionrdquoNeuroImage vol 54 no 2 pp 760ndash768 2011

Computational Intelligence and Neuroscience 15

[17] M B Cuadra L Cammoun T Butz O Cuisenaire and J-P Thiran ldquoComparison and validation of tissue modelizationand statistical classification methods in T1-weighted MR brainimagesrdquo IEEE Transactions on Medical Imaging vol 24 no 12pp 1548ndash1565 2005

[18] F Klauschen A Goldman V Barra A Meyer-Lindenbergand A Lundervold ldquoEvaluation of automated brain MR imagesegmentation and volumetrymethodsrdquoHuman BrainMappingvol 30 no 4 pp 1310ndash1327 2009

[19] H Zaidi T Ruest F Schoenahl and M-L Montandon ldquoCom-parative assessment of statistical brain MR image segmentationalgorithms and their impact on partial volume correction inPETrdquo NeuroImage vol 32 no 4 pp 1591ndash1607 2006

[20] J A Frazier S M Hodge J L Breeze et al ldquoDiagnostic and sexeffects on limbic volumes in early-onset bipolar disorder andschizophreniardquo Schizophrenia Bulletin vol 34 no 1 pp 37ndash462008

[21] A Klein and J Tourville ldquo101 labeled brain images and aconsistent human cortical labeling protocolrdquo Frontiers in Neu-roscience vol 6 article 171 2012

[22] B Van Ginneken T Heimann and M Styner ldquo3D segmen-tation in the clinic a grand challengerdquo in Proceedings of the10th International Conference on Medical Image Computingand Computer Assisted Intervention (MICCAI rsquo07) pp 7ndash15Brisbane Australia October-November 2007

[23] A M Dale B Fischl and M I Sereno ldquoCortical surface-based analysis I Segmentation and surface reconstructionrdquoNeuroImage vol 9 no 2 pp 179ndash194 1999

[24] B Fischl M I Sereno and A M Dale ldquoCortical surface-basedanalysis II inflation flattening and a surface-based coordinatesystemrdquo NeuroImage vol 9 no 2 pp 195ndash207 1999

[25] M Jenkinson C F Beckmann T E J BehrensMWWoolrichand S M Smith ldquoFSLrdquoNeuroImage vol 62 no 2 pp 782ndash7902012

[26] J Ashburner and K J Friston ldquoUnified segmentationrdquo Neu-roImage vol 26 no 3 pp 839ndash851 2005

[27] Y D Reijmer A Leemans M Brundel L J Kappelle and GJ Biessels ldquoDisruption of the cerebral white matter network isrelated to slowing of information processing speed in patientswith type 2 diabetesrdquoDiabetes vol 62 no 6 pp 2112ndash2115 2013

[28] S Klein M Staring K Murphy M A Viergever and J P WPluim ldquoElastix a toolbox for intensity-based medical imageregistrationrdquo IEEE Transactions onMedical Imaging vol 29 no1 pp 196ndash205 2010

[29] K J Friston J T Ashburner S J Kiebel T E Nichols and WD Penny Eds Statistical Parametric Mapping The Analysis ofFunctional Brain Images Academic Press 2011

[30] F Ritter T Boskamp A Homeyer et al ldquoMedical imageanalysis a visual approachrdquo IEEE Pulse vol 2 no 6 pp 60ndash702011

[31] L R Dice ldquoMeasures of the amount of ecologic associationbetween speciesrdquo Ecology vol 26 no 3 pp 297ndash302 1945

[32] D P Huttenlocher G A Klanderman and W J RucklidgeldquoComparing images using the Hausdorff distancerdquo IEEE Trans-actions on Pattern Analysis and Machine Intelligence vol 15 no9 pp 850ndash863 1993

[33] A Alansary A Soliman F Khalifa et al ldquoMAP-based frame-work for segmentation of MR brain images based on visualappearance and prior shaperdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal September 2013

[34] A Jog S Roy J L Prince and A Carass ldquoMR brain segmenta-tion using decision treesrdquo in Proceedings of the MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal Nagoya Japan September 2013

[35] R Katyal S Paneri and M Kuse ldquoGaussian intensity modelwith neighborhood cues for fluid-tissue categorization ofmulti-sequence MR brain imagesrdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal Nagoya Japan September 2013

[36] QMahmoodMAlipoor A Chodorowski AMehnert andMPersson ldquoMultimodal MR brain segmentation using Bayesian-based adaptive mean-shift (BAMS)rdquo in Proceedings of theMICCAI WorkshopsmdashThe MICCAI Grand Challenge on MRBrain Image Segmentation (MRBrainS rsquo13) The Midas JournalSeptember 2013

[37] A van Opbroek F van der Lijn andM de Bruijne ldquoAutomatedbrain-tissue segmentation by multi-feature SVM classificationrdquoin Proceedings of the MICCAI WorkshopsmdashThe MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal September 2013

[38] S Pereira J Festa J A Mariz N Sousa and C A SilvaldquoAutomatic brain tissue segmentation of multi-sequence MRimages using random decision forestsrdquo in Proceedings of theMICCAI Grand Challenge on MR Brain Image Segmentation(MRBrainS rsquo13) The Midas Journal Nagoya Japan September2013

[39] MRajchl J Baxter J Yuan T Peters andAKhan ldquoMulti-atlas-based segmentation with hierarchical max-flowrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[40] D Sarikaya L Zhao and J J Corso ldquoMulti-atlas brain MRIsegmentation with multiway cutrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[41] H Vrooman F van der Lijn and W Niessen ldquoAuto-kNNbrain tissue segmentation using automatically trained k-nearest-neighbor classificationrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[42] S Vyas P Burlina D Kleissas and R Mukherjee ldquoAutomatedwalks using machine learning for segmentationrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[43] C Wang and O Smedby ldquoFully automatic brain segmentationusing model-guided level sets and skeleton-based modelsrdquo inProceedings of theMICCAI Grand Challenge onMRBrain ImageSegmentation (MRBrainS rsquo13) The Midas Journal NagoyaJapan September 2013

[44] C Cortes and V Vapnik ldquoSupport-vector networksrdquo MachineLearning vol 20 no 3 pp 273ndash297 1995

[45] H A Vrooman C A Cocosco F van der Lijn et al ldquoMulti-spectral brain tissue segmentation using automatically trainedk-Nearest-Neighbor classificationrdquo NeuroImage vol 37 no 1pp 71ndash81 2007

[46] C Wang H Frimmel and O Smedby ldquoFast level-set basedimage segmentation using coherent propagationrdquo MedicalPhysics vol 41 no 7 Article ID 073501 2014

16 Computational Intelligence and Neuroscience

[47] A Jog S Roy A Carass and J L Prince ldquoMagnetic resonanceimage synthesis through patch regressionrdquo in Proceedings of theIEEE 10th International Symposium onBiomedical Imaging (ISBIrsquo13) pp 350ndash353 April 2013

[48] J G Sied A P Zijdenbos and A C Evans ldquoA non-parametricmethod for automatic correction of intensity non-uniformity inMRI datardquo IEEE Transactions on Medical Imaging vol 17 no 1pp 87ndash97 1998

[49] A Carass J Cuzzocreo M B Wheeler P-L Bazin S MResnick and J L Prince ldquoSimple paradigm for extra-cerebraltissue removal algorithm and analysisrdquoNeuroImage vol 56 no4 pp 1982ndash1992 2011

[50] M Modat G R Ridgway Z A Taylor et al ldquoFast free-form deformation using graphics processing unitsrdquo ComputerMethods and Programs in Biomedicine vol 98 no 3 pp 278ndash284 2010

[51] M Rajchl J Yuan J A White et al ldquoInteractive hierarchical-flow segmentation of scar tissue from late-enhancement cardiacmr imagesrdquo IEEE Transactions on Medical Imaging vol 33 no1 pp 159ndash172 2014

[52] QMahmood A Chodorowski AMehnert andM Persson ldquoAnovel Bayesian approach to adaptive mean shift segmentationof brain imagesrdquo in Proceedings of the 25th IEEE InternationalSymposiumonComputer-BasedMedical Systems (CBMS rsquo12) pp1ndash6 June 2012

[53] A El-Baz M Nitzken A Alansary A Soliman and MCasanova ldquoBrain Segmentation Method for Young Childrenand Adultsrdquo US Provisional Patent Application 13100

[54] A Alansary A Soliman M Nitzken et al ldquoAn integratedgeometrical and stochastic approach for accurate infant brainextractionrdquo in Proceedings of the IEEE International Conferenceon Image Processing (ICIP rsquo14) pp 3542ndash3546 October 2014

[55] V Popescu M Battaglini W S Hoogstrate et al ldquoOptimizingparameter choice for FSL-Brain Extraction Tool (BET) on 3DT1 images in multiple sclerosisrdquo NeuroImage vol 61 no 4 pp1484ndash1494 2012

[56] J L Whitwell W R Crum H C Watt and N C FoxldquoNormalization of cerebral volumes by use of intracranial vol-ume implications for longitudinal quantitative MR imagingrdquoAmerican Journal of Neuroradiology vol 22 no 8 pp 1483ndash1489 2001

[57] S M Smith ldquoFast robust automated brain extractionrdquo HumanBrain Mapping vol 17 no 3 pp 143ndash155 2002

[58] D W Shattuck G Prasad M Mirza K L Narr and A WToga ldquoOnline resource for validation of brain segmentationmethodsrdquo NeuroImage vol 45 no 2 pp 431ndash439 2009

[59] I Isgum M J N L Benders B Avants et al ldquoEvaluation ofautomatic neonatal brain segmentation algorithms the Neo-BrainS12 challengerdquo Medical Image Analysis vol 20 no 1 pp135ndash151 2015

[60] P Moeskops M J N L Benders S M Chita et al ldquoAutomaticsegmentation of MR brain images of preterm infants usingsupervised classificationrdquo NeuroImage 2015

[61] P Moeskops M A Viergever M J N L Benders and IIsgum ldquoEvaluation of an automatic brain segmentationmethoddeveloped for neonates on adult MR brain imagesrdquo in MedicalImaging Image Processing vol 9413 of Proceedings of SPIEInternational Society for Optics and Photonics Orlando FlaUSA February 2015

[62] L Wang Y Gao F Shi et al ldquoLINKS learning-based multi-source IntegratioN frameworK for Segmentation of infant brainimagesrdquo NeuroImage vol 108 pp 160ndash172 2015

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

8 Computational Intelligence and Neuroscience

Table 1 Continued

119903 Team 119905 Scans

GM WM CSF

119904

Brain ICV119903001199031011990320119903011199031111990321119903021199031211990322

119903031199031311990323119903041199031411990324

119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD 119863 11986795

AVD120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590) 120583 (120590)

13 SPM T1 IR F 4m T1 IR F10 12 18 15 14 12 7 11 6

10512 15 13 3 15 3

801 30 139 836 38 84 769 41 12 936 59 51 977 82 12(24) (02) (96) (21) (05) (52) (31) (05) (60) (11) (18) (36) (02) (18) (09)

14 FSL PVSeg 10m T115 15 8 13 15 17 13 7 4

10711 16 9 13 7 7

777 43 84 848 38 197 695 34 11 936 61 35 942 53 34(26) (13) (65) (32) (09) (10) (22) (03) (58) (09) (16) (30) (08) (11) (15)

15 FreeSurfer 1 h 3D T116 6 17 12 9 8 18 12 18

11617 6 16 17 10 18

774 23 121 852 31 72 658 43 50 923 36 63 924 66 10(20) (06) (60) (22) (05) (46) (37) (06) (196) (08) (06) (34) (09) (04) (45)

16 Jedi MindMeld 27 s T1 IR F

14 14 11 17 16 15 10 8 11116

15 12 8 11 16 14778 39 89 806 45 137 733 37 18 932 54 35 943 20 68(59) (28) (76) (96) (36) (12) (35) (08) (90) (17) (48) (30) (17) (37) (31)

17 S2 QM 15 h T1 IR F17 17 13 14 17 18 16 9 9

13014 17 15 14 9 10

764 55 93 839 49 248 679 38 14 933 70 55 937 65 40(34) (30) (64) (34) (32) (10) (23) (06) (99) (14) (35) (45) (11) (14) (22)

18 LNMBrains 5m T1 IR F18 18 14 18 18 16 14 17 12

14518 18 17 18 17 16

728 68 95 783 68 153 688 77 20 885 86 74 892 204 79(53) (23) (76) (64) (35) (13) (66) (24) (13) (45) (28) (77) (48) (37) (83)

lowastSemiautomatic due to per scan parameter tuning

Besides the advantages of the MRBrainS evaluationframework there are some limitations that should be takeninto account The T1-weighted IR and the T2-weightedFLAIR scan were acquired with a lower resolution (096times 096 times 300mm3) than the 3D T1-weighted scan (10 times10 times 10mm3) To be able to provide a registered multise-quence dataset the 3D T1-weighted scan was registered tothe T2-weighted FLAIR scan and downsampled to 096 times096 times 300mm3 The reference standard is therefore onlyavailable for this resolution The decreased performance ofthe FreeSurfer GM segmentation as compared to the otherfreeware packages might be due to the fact that we evaluateon the thick slice T1 sequence instead of the high resolutionT1 Performing the manual segmentations to provide thereference standard is very laborsome and time consumingInstead of letting multiple observers manually segment theMRI datasets or letting one observer manually segmentthe MRI datasets twice much time and effort was spenton creating one reference standard that was as accurate aspossible Therefore we were not able to determine the inter-or intraobserver variability Finally we acknowledge that ourevaluation framework is limited to evaluating the accuracyand robustness over 15 datasets for segmenting GM WMand CSF on 3T MRI scans acquired on a Philips scanner ofa specific group of elderly subjects Many factors influencesegmentation algorithm performance such as the type ofscanner (vendor field strength) the acquisition protocolthe available MRI sequences and the type of subjects

Participating algorithmsmight have been designed for differ-ent types of MRI scans Therefore the five provided trainingdatasets are important for participants to be able to traintheir algorithms on the provided data Some algorithms aredesigned to segment only some components such as onlyGM and WM instead of all three components and usefreely available software such as the brain extraction tool[57] to segment the outer border of the CSF (intracranialvolume) We have chosen to base the final ranking on allthree components but it is therefore important to assess notonly the final ranking but the performance of the individualcomponents as well

Despite these limitations the MRBrainS evaluationframework provides an objective and direct comparison ofsegmentation algorithms The reference standard of the testdata is unknown to the participants the same evaluationmeasures are used for all evaluated algorithms and partici-pants apply their own algorithms to the provided data

In comparison to the online validation engine proposedby Shattuck et al [58] the MRBrainS evaluation frameworkuses 3T MRI data instead of 15T MRI data and evaluates notonly brain versus nonbrain segmentation but also segmenta-tion of gray matter white matter cerebrospinal fluid brainand intracranial volume The availability of many differenttypes of evaluation frameworks will aid in the developmentof more generic and robust algorithms For example inthe NEATBrainS (httpneatbrains15isiuunl) challengeresearchers were challenged to apply their algorithms to data

Computational Intelligence and Neuroscience 9

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

65

70

75

80

85

GM Dice

Dic

e (

)

(a)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

55

60

65

70

75

80

85

90

WM Dice

Dic

e (

)

(b)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

55

60

65

70

75

80

85CSF Dice

Dic

e (

)

(c)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

80

85

90

95

Brain (WM + GM) Dice

Dic

e (

)

(d)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

80

85

90

95

Dic

e (

)

Intracranial volume (WM + GM + CSF) Dice

(e)

Figure 2 Boxplots presenting the evaluation results for the Dice coefficient (1) of the gray matter (GM) white matter (WM) cerebrospinalfluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all 15 test datasets

10 Computational Intelligence and Neuroscience

2468

10121416

GM H-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(a)

2468

10121416

WMH-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(b)

5

10

15

20CSF H-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(c)

5

10

15

20H

-95

(mm

)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Brain (WM + GM) H-95

(d)

5

10

15

20

25

30

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Intracranial volume (WM + GM + CSF) H-95

(e)

Figure 3 Boxplots presenting the evaluation results for the 95th-percentile Hausdorff distance (3) of the gray matter (GM) white matter(WM) cerebrospinal fluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all15 test datasets

Computational Intelligence and Neuroscience 11

0

10

20

30

GM AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(a)

0

10

20

30

40WM AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(b)

0

20

40

60

80

CSF AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(c)

0

5

10

15

20

25

30AV

D (

)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Brain (WM + GM) AVD

(d)

0

5

10

15

20

25

30

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Intracranial volume (WM + GM + CSF) AVD

(e)

Figure 4 Boxplots presenting the evaluation results for the absolute volume difference (4) of the gray matter (GM) white matter (WM)cerebrospinal fluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all 15 testdatasets

12 Computational Intelligence and Neuroscience

T2

UofL

N

FS

S2

‐FLAIR

BioImaging

Narsil

L‐Seg

2_QM

T1‐IR

CMIV

SPM12_T1

SPM12_T1_

LNMBr

_FLAIR

_IR_FLAIR

ains

T1

UB VPML Me

SPM12_T1_ I

FSL‐PVSeg

R

d B

R

F

eference

igr_neuro

MNAB

reeSurfer

BIG

Rob

SPM1

Jedi Mi

R2

arts

2_T1

nd Meld

Figure 5 Illustration of the segmentation results at the height of the basal ganglia (test subject 9 slice 22) The basal ganglia should besegmented as gray matter The first three images show the three MRI sequences The fourth image (reference) is the manually segmentedreference standard (yellow white matter light blue gray matter and dark blue cerebrospinal fluid) The results of the participatingsegmentation algorithms are ordered from the best overall performance (BIGR2) to the worst overall performance (LNMBrains) Majordifferences are present in the segmentation of the basal ganglia and the outer border of the cerebrospinal fluid The arrows indicate examplelocations where the segmentation results differ from the ground truth

Computational Intelligence and Neuroscience 13

T2

UofL BioIm

Nars

FSL‐S

S2_Q

‐FLAIR

aging (860)

il (892)

eg (453)

M (565)

T1‐IR

CMIV (90

SPM12_T1_F

SPM12_T1_IR

LNMBrains

5) UB

(831) SP

_F (815) F

(286)

T1

VPML Med (9

M12_T1_IR (60

SL‐PVSeg (525

R

10) Bigr_

5) MN

) FreeS

eference

neuro (896)

AB (889)

urfer (846)

BIGR2

Robarts

SPM12_T

Jedi Mind M

(852)

(810)

1 (683)

eld (769)

Figure 6 Illustration of the segmentation results in the presence of whitematter lesions (test subject 3 slice 31)Whitematter lesions (WMLs)should be labeled asWM the sensitivity (percentage ofWML voxels is labeled asWM) over all 15 datasets is presented between brackets afterthe team nameThe first three images show the threeMRI sequences T2-weighted fluid attenuated inversion recovery T1-weighted inversionrecovery and T1-weighted scanThe fourth image (reference) is themanually segmented reference standard (red whitematter lesions yellowwhite matter light blue gray matter and dark blue cerebrospinal fluid)The results of the segmentation algorithms are ordered from the bestoverall performance (BIGR2) to the worst overall performance (LNMBrains)The arrows indicate example locations where the segmentationresults differ from the ground truth

14 Computational Intelligence and Neuroscience

fromboth theMRBrainS and theNeoBrainS [59] (brain tissuesegmentation in neonates) challenge Two methods [60ndash62]specifically designed for neonatal brain tissue segmentationshowed a high performance for tissue segmentation on theMRBrainS data Applying algorithms to different types of datahas the potential to lead to new insights and more robustalgorithms The MRBrainS evaluation framework remainsopen for new contributions At the time of writing 21teams had submitted their results on the MRBrainS website(httpmrbrains13isiuunlresultsphp)

5 Conclusion

The MRBrainS challenge online evaluation framework pro-vides direct and objective comparison of automatic andsemiautomatic methods to segment GM WM CSF brainand ICV on 3T multisequence MRI data The first elevenparticipating methods are evaluated and presented in thispaper as well as three commonly used freeware packages(FreeSurfer FSL and SPM12) They provide a benchmarkfor future contributions to the framework The final rankingprovides a quick insight into the overall performance of theevaluated methods in comparison to each other whereasthe individual evaluation measures (Dice 95th-percentileHausdorff distance and absolute volume difference) percomponent (GM WM CSF brain and ICV) can aid inselecting the best method for a specific segmentation goal

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was financially supported by IMDI Grant104002002 (Brainbox) fromZonMw the Netherlands Organ-isation for Health Research and Development within kindsponsoring by Philips theUniversityMedical CenterUtrechtand Eindhoven University of TechnologyThe authors wouldlike to acknowledge the following members of the UtrechtVascular Cognitive Impairment Study Group who were notincluded as coauthors of this paper but were involved in therecruitment of study participants and MRI acquisition at theUMC Utrecht (in alphabetical order by department) E vanden Berg M Brundel S Heringa and L J Kappelle of theDepartment of Neurology P R Luijten and W P Th MMali of the Department of Radiology and A Algra and GE H M Rutten of the Julius Center for Health Sciences andPrimary CareThe research of Geert Jan Biessels and the VCIgroupwas financially supported byVIDIGrant 91711384 fromZonMw and by Grant 2010T073 of the Netherlands HeartFoundation The research of Jeroen de Bresser is financiallysupported by a research talent fellowship of the Univer-sity Medical Center Utrecht (Netherlands) The research ofAnnegreet vanOpbroek andMarleen de Bruijne is financiallysupported by a research grant from NWO (the NetherlandsOrganisation for Scientific Research) The authors would

like to acknowledge MeVis Medical Solutions AG (BremenGermany) for providing MeVisLab Duygu Sarikaya andLiang Zhao acknowledge their Advisor Professor JasonCorsofor his guidance Duygu Sarikaya is supported by NIH 1 R21CA160825-01 and Liang Zhao is partially supported by theChina Scholarship Council (CSC)

References

[1] I Driscoll C Davatzikos Y An et al ldquoLongitudinal pattern ofregional brain volume change differentiates normal aging fromMCIrdquo Neurology vol 72 no 22 pp 1906ndash1913 2009

[2] K Franke and C Gaser ldquoLongitudinal changes in individualBrainAGE in healthy aging mild cognitive impairment andAlzheimerrsquos diseaserdquoGeroPsych vol 25 no 4 pp 235ndash245 2012

[3] MA IkramH A VroomanMWVernooij et al ldquoBrain tissuevolumes in the general elderly populationThe Rotterdam ScanStudyrdquo Neurobiology of Aging vol 29 no 6 pp 882ndash890 2008

[4] A Giorgio and N de Stefano ldquoClinical use of brain volumetryrdquoJournal of Magnetic Resonance Imaging vol 37 no 1 pp 1ndash142013

[5] M W Vannier R L Butterfield D Jordan W A Murphy RG Levitt and M Gado ldquoMultispectral analysis of magneticresonance imagesrdquo Radiology vol 154 no 1 pp 221ndash224 1985

[6] M A Balafar A R Ramli M I Saripan and S MashohorldquoReview of brain MRI image segmentation methodsrdquo ArtificialIntelligence Review vol 33 no 3 pp 261ndash274 2010

[7] J C Bezdek L O Hall and L P Clarke ldquoReview of MR imagesegmentation techniques using pattern recognitionrdquo MedicalPhysics vol 20 no 4 pp 1033ndash1048 1993

[8] L P Clarke R P Velthuizen M A Camacho et al ldquoMRIsegmentation methods and applicationsrdquo Magnetic ResonanceImaging vol 13 no 3 pp 343ndash368 1995

[9] J S Duncan X Papademetris J Yang M Jackowski X Zengand L H Staib ldquoGeometric strategies for neuroanatomic ana-lysis from MRIrdquo NeuroImage vol 23 supplement 1 pp S34ndashS45 2004

[10] D L PhamC Xu and J L Prince ldquoCurrentmethods inmedicalimage segmentationrdquoAnnual Review of Biomedical Engineeringvol 2 no 2000 pp 315ndash337 2000

[11] N Saeed ldquoMagnetic resonance image segmentation usingpattern recognition and applied to image registration andquantitationrdquo NMR in Biomedicine vol 11 no 4-5 pp 157ndash1671998

[12] J S Suri S Singh and L Reden ldquoComputer vision andpattern recognition techniques for 2-D and 3-D MR cerebralcortical segmentation (Part I) a state-of-the-art reviewrdquo PatternAnalysis amp Applications vol 5 no 1 pp 46ndash76 2002

[13] A P Zijdenbos and B M Dawant ldquoBrain segmentation andwhite matter lesion detection in MR imagesrdquo Critical Reviewsin Biomedical Engineering vol 22 no 5-6 pp 401ndash465 1994

[14] K Price ldquoAnything you can do I can do better (No you canrsquot)rdquoComputer Vision Graphics and Image Processing vol 36 no 2-3pp 387ndash391 1986

[15] R de Boer H A Vrooman M A Ikram et al ldquoAccuracy andreproducibility study of automatic MRI brain tissue segmenta-tion methodsrdquo NeuroImage vol 51 no 3 pp 1047ndash1056 2010

[16] J de Bresser M P Portegies A Leemans G J Biessels L JKappelle andM A Viergever ldquoA comparison ofMR based seg-mentation methods for measuring brain atrophy progressionrdquoNeuroImage vol 54 no 2 pp 760ndash768 2011

Computational Intelligence and Neuroscience 15

[17] M B Cuadra L Cammoun T Butz O Cuisenaire and J-P Thiran ldquoComparison and validation of tissue modelizationand statistical classification methods in T1-weighted MR brainimagesrdquo IEEE Transactions on Medical Imaging vol 24 no 12pp 1548ndash1565 2005

[18] F Klauschen A Goldman V Barra A Meyer-Lindenbergand A Lundervold ldquoEvaluation of automated brain MR imagesegmentation and volumetrymethodsrdquoHuman BrainMappingvol 30 no 4 pp 1310ndash1327 2009

[19] H Zaidi T Ruest F Schoenahl and M-L Montandon ldquoCom-parative assessment of statistical brain MR image segmentationalgorithms and their impact on partial volume correction inPETrdquo NeuroImage vol 32 no 4 pp 1591ndash1607 2006

[20] J A Frazier S M Hodge J L Breeze et al ldquoDiagnostic and sexeffects on limbic volumes in early-onset bipolar disorder andschizophreniardquo Schizophrenia Bulletin vol 34 no 1 pp 37ndash462008

[21] A Klein and J Tourville ldquo101 labeled brain images and aconsistent human cortical labeling protocolrdquo Frontiers in Neu-roscience vol 6 article 171 2012

[22] B Van Ginneken T Heimann and M Styner ldquo3D segmen-tation in the clinic a grand challengerdquo in Proceedings of the10th International Conference on Medical Image Computingand Computer Assisted Intervention (MICCAI rsquo07) pp 7ndash15Brisbane Australia October-November 2007

[23] A M Dale B Fischl and M I Sereno ldquoCortical surface-based analysis I Segmentation and surface reconstructionrdquoNeuroImage vol 9 no 2 pp 179ndash194 1999

[24] B Fischl M I Sereno and A M Dale ldquoCortical surface-basedanalysis II inflation flattening and a surface-based coordinatesystemrdquo NeuroImage vol 9 no 2 pp 195ndash207 1999

[25] M Jenkinson C F Beckmann T E J BehrensMWWoolrichand S M Smith ldquoFSLrdquoNeuroImage vol 62 no 2 pp 782ndash7902012

[26] J Ashburner and K J Friston ldquoUnified segmentationrdquo Neu-roImage vol 26 no 3 pp 839ndash851 2005

[27] Y D Reijmer A Leemans M Brundel L J Kappelle and GJ Biessels ldquoDisruption of the cerebral white matter network isrelated to slowing of information processing speed in patientswith type 2 diabetesrdquoDiabetes vol 62 no 6 pp 2112ndash2115 2013

[28] S Klein M Staring K Murphy M A Viergever and J P WPluim ldquoElastix a toolbox for intensity-based medical imageregistrationrdquo IEEE Transactions onMedical Imaging vol 29 no1 pp 196ndash205 2010

[29] K J Friston J T Ashburner S J Kiebel T E Nichols and WD Penny Eds Statistical Parametric Mapping The Analysis ofFunctional Brain Images Academic Press 2011

[30] F Ritter T Boskamp A Homeyer et al ldquoMedical imageanalysis a visual approachrdquo IEEE Pulse vol 2 no 6 pp 60ndash702011

[31] L R Dice ldquoMeasures of the amount of ecologic associationbetween speciesrdquo Ecology vol 26 no 3 pp 297ndash302 1945

[32] D P Huttenlocher G A Klanderman and W J RucklidgeldquoComparing images using the Hausdorff distancerdquo IEEE Trans-actions on Pattern Analysis and Machine Intelligence vol 15 no9 pp 850ndash863 1993

[33] A Alansary A Soliman F Khalifa et al ldquoMAP-based frame-work for segmentation of MR brain images based on visualappearance and prior shaperdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal September 2013

[34] A Jog S Roy J L Prince and A Carass ldquoMR brain segmenta-tion using decision treesrdquo in Proceedings of the MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal Nagoya Japan September 2013

[35] R Katyal S Paneri and M Kuse ldquoGaussian intensity modelwith neighborhood cues for fluid-tissue categorization ofmulti-sequence MR brain imagesrdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal Nagoya Japan September 2013

[36] QMahmoodMAlipoor A Chodorowski AMehnert andMPersson ldquoMultimodal MR brain segmentation using Bayesian-based adaptive mean-shift (BAMS)rdquo in Proceedings of theMICCAI WorkshopsmdashThe MICCAI Grand Challenge on MRBrain Image Segmentation (MRBrainS rsquo13) The Midas JournalSeptember 2013

[37] A van Opbroek F van der Lijn andM de Bruijne ldquoAutomatedbrain-tissue segmentation by multi-feature SVM classificationrdquoin Proceedings of the MICCAI WorkshopsmdashThe MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal September 2013

[38] S Pereira J Festa J A Mariz N Sousa and C A SilvaldquoAutomatic brain tissue segmentation of multi-sequence MRimages using random decision forestsrdquo in Proceedings of theMICCAI Grand Challenge on MR Brain Image Segmentation(MRBrainS rsquo13) The Midas Journal Nagoya Japan September2013

[39] MRajchl J Baxter J Yuan T Peters andAKhan ldquoMulti-atlas-based segmentation with hierarchical max-flowrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[40] D Sarikaya L Zhao and J J Corso ldquoMulti-atlas brain MRIsegmentation with multiway cutrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[41] H Vrooman F van der Lijn and W Niessen ldquoAuto-kNNbrain tissue segmentation using automatically trained k-nearest-neighbor classificationrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[42] S Vyas P Burlina D Kleissas and R Mukherjee ldquoAutomatedwalks using machine learning for segmentationrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[43] C Wang and O Smedby ldquoFully automatic brain segmentationusing model-guided level sets and skeleton-based modelsrdquo inProceedings of theMICCAI Grand Challenge onMRBrain ImageSegmentation (MRBrainS rsquo13) The Midas Journal NagoyaJapan September 2013

[44] C Cortes and V Vapnik ldquoSupport-vector networksrdquo MachineLearning vol 20 no 3 pp 273ndash297 1995

[45] H A Vrooman C A Cocosco F van der Lijn et al ldquoMulti-spectral brain tissue segmentation using automatically trainedk-Nearest-Neighbor classificationrdquo NeuroImage vol 37 no 1pp 71ndash81 2007

[46] C Wang H Frimmel and O Smedby ldquoFast level-set basedimage segmentation using coherent propagationrdquo MedicalPhysics vol 41 no 7 Article ID 073501 2014

16 Computational Intelligence and Neuroscience

[47] A Jog S Roy A Carass and J L Prince ldquoMagnetic resonanceimage synthesis through patch regressionrdquo in Proceedings of theIEEE 10th International Symposium onBiomedical Imaging (ISBIrsquo13) pp 350ndash353 April 2013

[48] J G Sied A P Zijdenbos and A C Evans ldquoA non-parametricmethod for automatic correction of intensity non-uniformity inMRI datardquo IEEE Transactions on Medical Imaging vol 17 no 1pp 87ndash97 1998

[49] A Carass J Cuzzocreo M B Wheeler P-L Bazin S MResnick and J L Prince ldquoSimple paradigm for extra-cerebraltissue removal algorithm and analysisrdquoNeuroImage vol 56 no4 pp 1982ndash1992 2011

[50] M Modat G R Ridgway Z A Taylor et al ldquoFast free-form deformation using graphics processing unitsrdquo ComputerMethods and Programs in Biomedicine vol 98 no 3 pp 278ndash284 2010

[51] M Rajchl J Yuan J A White et al ldquoInteractive hierarchical-flow segmentation of scar tissue from late-enhancement cardiacmr imagesrdquo IEEE Transactions on Medical Imaging vol 33 no1 pp 159ndash172 2014

[52] QMahmood A Chodorowski AMehnert andM Persson ldquoAnovel Bayesian approach to adaptive mean shift segmentationof brain imagesrdquo in Proceedings of the 25th IEEE InternationalSymposiumonComputer-BasedMedical Systems (CBMS rsquo12) pp1ndash6 June 2012

[53] A El-Baz M Nitzken A Alansary A Soliman and MCasanova ldquoBrain Segmentation Method for Young Childrenand Adultsrdquo US Provisional Patent Application 13100

[54] A Alansary A Soliman M Nitzken et al ldquoAn integratedgeometrical and stochastic approach for accurate infant brainextractionrdquo in Proceedings of the IEEE International Conferenceon Image Processing (ICIP rsquo14) pp 3542ndash3546 October 2014

[55] V Popescu M Battaglini W S Hoogstrate et al ldquoOptimizingparameter choice for FSL-Brain Extraction Tool (BET) on 3DT1 images in multiple sclerosisrdquo NeuroImage vol 61 no 4 pp1484ndash1494 2012

[56] J L Whitwell W R Crum H C Watt and N C FoxldquoNormalization of cerebral volumes by use of intracranial vol-ume implications for longitudinal quantitative MR imagingrdquoAmerican Journal of Neuroradiology vol 22 no 8 pp 1483ndash1489 2001

[57] S M Smith ldquoFast robust automated brain extractionrdquo HumanBrain Mapping vol 17 no 3 pp 143ndash155 2002

[58] D W Shattuck G Prasad M Mirza K L Narr and A WToga ldquoOnline resource for validation of brain segmentationmethodsrdquo NeuroImage vol 45 no 2 pp 431ndash439 2009

[59] I Isgum M J N L Benders B Avants et al ldquoEvaluation ofautomatic neonatal brain segmentation algorithms the Neo-BrainS12 challengerdquo Medical Image Analysis vol 20 no 1 pp135ndash151 2015

[60] P Moeskops M J N L Benders S M Chita et al ldquoAutomaticsegmentation of MR brain images of preterm infants usingsupervised classificationrdquo NeuroImage 2015

[61] P Moeskops M A Viergever M J N L Benders and IIsgum ldquoEvaluation of an automatic brain segmentationmethoddeveloped for neonates on adult MR brain imagesrdquo in MedicalImaging Image Processing vol 9413 of Proceedings of SPIEInternational Society for Optics and Photonics Orlando FlaUSA February 2015

[62] L Wang Y Gao F Shi et al ldquoLINKS learning-based multi-source IntegratioN frameworK for Segmentation of infant brainimagesrdquo NeuroImage vol 108 pp 160ndash172 2015

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience 9

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

65

70

75

80

85

GM Dice

Dic

e (

)

(a)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

55

60

65

70

75

80

85

90

WM Dice

Dic

e (

)

(b)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

55

60

65

70

75

80

85CSF Dice

Dic

e (

)

(c)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

80

85

90

95

Brain (WM + GM) Dice

Dic

e (

)

(d)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

80

85

90

95

Dic

e (

)

Intracranial volume (WM + GM + CSF) Dice

(e)

Figure 2 Boxplots presenting the evaluation results for the Dice coefficient (1) of the gray matter (GM) white matter (WM) cerebrospinalfluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all 15 test datasets

10 Computational Intelligence and Neuroscience

2468

10121416

GM H-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(a)

2468

10121416

WMH-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(b)

5

10

15

20CSF H-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(c)

5

10

15

20H

-95

(mm

)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Brain (WM + GM) H-95

(d)

5

10

15

20

25

30

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Intracranial volume (WM + GM + CSF) H-95

(e)

Figure 3 Boxplots presenting the evaluation results for the 95th-percentile Hausdorff distance (3) of the gray matter (GM) white matter(WM) cerebrospinal fluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all15 test datasets

Computational Intelligence and Neuroscience 11

0

10

20

30

GM AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(a)

0

10

20

30

40WM AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(b)

0

20

40

60

80

CSF AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(c)

0

5

10

15

20

25

30AV

D (

)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Brain (WM + GM) AVD

(d)

0

5

10

15

20

25

30

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Intracranial volume (WM + GM + CSF) AVD

(e)

Figure 4 Boxplots presenting the evaluation results for the absolute volume difference (4) of the gray matter (GM) white matter (WM)cerebrospinal fluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all 15 testdatasets

12 Computational Intelligence and Neuroscience

T2

UofL

N

FS

S2

‐FLAIR

BioImaging

Narsil

L‐Seg

2_QM

T1‐IR

CMIV

SPM12_T1

SPM12_T1_

LNMBr

_FLAIR

_IR_FLAIR

ains

T1

UB VPML Me

SPM12_T1_ I

FSL‐PVSeg

R

d B

R

F

eference

igr_neuro

MNAB

reeSurfer

BIG

Rob

SPM1

Jedi Mi

R2

arts

2_T1

nd Meld

Figure 5 Illustration of the segmentation results at the height of the basal ganglia (test subject 9 slice 22) The basal ganglia should besegmented as gray matter The first three images show the three MRI sequences The fourth image (reference) is the manually segmentedreference standard (yellow white matter light blue gray matter and dark blue cerebrospinal fluid) The results of the participatingsegmentation algorithms are ordered from the best overall performance (BIGR2) to the worst overall performance (LNMBrains) Majordifferences are present in the segmentation of the basal ganglia and the outer border of the cerebrospinal fluid The arrows indicate examplelocations where the segmentation results differ from the ground truth

Computational Intelligence and Neuroscience 13

T2

UofL BioIm

Nars

FSL‐S

S2_Q

‐FLAIR

aging (860)

il (892)

eg (453)

M (565)

T1‐IR

CMIV (90

SPM12_T1_F

SPM12_T1_IR

LNMBrains

5) UB

(831) SP

_F (815) F

(286)

T1

VPML Med (9

M12_T1_IR (60

SL‐PVSeg (525

R

10) Bigr_

5) MN

) FreeS

eference

neuro (896)

AB (889)

urfer (846)

BIGR2

Robarts

SPM12_T

Jedi Mind M

(852)

(810)

1 (683)

eld (769)

Figure 6 Illustration of the segmentation results in the presence of whitematter lesions (test subject 3 slice 31)Whitematter lesions (WMLs)should be labeled asWM the sensitivity (percentage ofWML voxels is labeled asWM) over all 15 datasets is presented between brackets afterthe team nameThe first three images show the threeMRI sequences T2-weighted fluid attenuated inversion recovery T1-weighted inversionrecovery and T1-weighted scanThe fourth image (reference) is themanually segmented reference standard (red whitematter lesions yellowwhite matter light blue gray matter and dark blue cerebrospinal fluid)The results of the segmentation algorithms are ordered from the bestoverall performance (BIGR2) to the worst overall performance (LNMBrains)The arrows indicate example locations where the segmentationresults differ from the ground truth

14 Computational Intelligence and Neuroscience

fromboth theMRBrainS and theNeoBrainS [59] (brain tissuesegmentation in neonates) challenge Two methods [60ndash62]specifically designed for neonatal brain tissue segmentationshowed a high performance for tissue segmentation on theMRBrainS data Applying algorithms to different types of datahas the potential to lead to new insights and more robustalgorithms The MRBrainS evaluation framework remainsopen for new contributions At the time of writing 21teams had submitted their results on the MRBrainS website(httpmrbrains13isiuunlresultsphp)

5 Conclusion

The MRBrainS challenge online evaluation framework pro-vides direct and objective comparison of automatic andsemiautomatic methods to segment GM WM CSF brainand ICV on 3T multisequence MRI data The first elevenparticipating methods are evaluated and presented in thispaper as well as three commonly used freeware packages(FreeSurfer FSL and SPM12) They provide a benchmarkfor future contributions to the framework The final rankingprovides a quick insight into the overall performance of theevaluated methods in comparison to each other whereasthe individual evaluation measures (Dice 95th-percentileHausdorff distance and absolute volume difference) percomponent (GM WM CSF brain and ICV) can aid inselecting the best method for a specific segmentation goal

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was financially supported by IMDI Grant104002002 (Brainbox) fromZonMw the Netherlands Organ-isation for Health Research and Development within kindsponsoring by Philips theUniversityMedical CenterUtrechtand Eindhoven University of TechnologyThe authors wouldlike to acknowledge the following members of the UtrechtVascular Cognitive Impairment Study Group who were notincluded as coauthors of this paper but were involved in therecruitment of study participants and MRI acquisition at theUMC Utrecht (in alphabetical order by department) E vanden Berg M Brundel S Heringa and L J Kappelle of theDepartment of Neurology P R Luijten and W P Th MMali of the Department of Radiology and A Algra and GE H M Rutten of the Julius Center for Health Sciences andPrimary CareThe research of Geert Jan Biessels and the VCIgroupwas financially supported byVIDIGrant 91711384 fromZonMw and by Grant 2010T073 of the Netherlands HeartFoundation The research of Jeroen de Bresser is financiallysupported by a research talent fellowship of the Univer-sity Medical Center Utrecht (Netherlands) The research ofAnnegreet vanOpbroek andMarleen de Bruijne is financiallysupported by a research grant from NWO (the NetherlandsOrganisation for Scientific Research) The authors would

like to acknowledge MeVis Medical Solutions AG (BremenGermany) for providing MeVisLab Duygu Sarikaya andLiang Zhao acknowledge their Advisor Professor JasonCorsofor his guidance Duygu Sarikaya is supported by NIH 1 R21CA160825-01 and Liang Zhao is partially supported by theChina Scholarship Council (CSC)

References

[1] I Driscoll C Davatzikos Y An et al ldquoLongitudinal pattern ofregional brain volume change differentiates normal aging fromMCIrdquo Neurology vol 72 no 22 pp 1906ndash1913 2009

[2] K Franke and C Gaser ldquoLongitudinal changes in individualBrainAGE in healthy aging mild cognitive impairment andAlzheimerrsquos diseaserdquoGeroPsych vol 25 no 4 pp 235ndash245 2012

[3] MA IkramH A VroomanMWVernooij et al ldquoBrain tissuevolumes in the general elderly populationThe Rotterdam ScanStudyrdquo Neurobiology of Aging vol 29 no 6 pp 882ndash890 2008

[4] A Giorgio and N de Stefano ldquoClinical use of brain volumetryrdquoJournal of Magnetic Resonance Imaging vol 37 no 1 pp 1ndash142013

[5] M W Vannier R L Butterfield D Jordan W A Murphy RG Levitt and M Gado ldquoMultispectral analysis of magneticresonance imagesrdquo Radiology vol 154 no 1 pp 221ndash224 1985

[6] M A Balafar A R Ramli M I Saripan and S MashohorldquoReview of brain MRI image segmentation methodsrdquo ArtificialIntelligence Review vol 33 no 3 pp 261ndash274 2010

[7] J C Bezdek L O Hall and L P Clarke ldquoReview of MR imagesegmentation techniques using pattern recognitionrdquo MedicalPhysics vol 20 no 4 pp 1033ndash1048 1993

[8] L P Clarke R P Velthuizen M A Camacho et al ldquoMRIsegmentation methods and applicationsrdquo Magnetic ResonanceImaging vol 13 no 3 pp 343ndash368 1995

[9] J S Duncan X Papademetris J Yang M Jackowski X Zengand L H Staib ldquoGeometric strategies for neuroanatomic ana-lysis from MRIrdquo NeuroImage vol 23 supplement 1 pp S34ndashS45 2004

[10] D L PhamC Xu and J L Prince ldquoCurrentmethods inmedicalimage segmentationrdquoAnnual Review of Biomedical Engineeringvol 2 no 2000 pp 315ndash337 2000

[11] N Saeed ldquoMagnetic resonance image segmentation usingpattern recognition and applied to image registration andquantitationrdquo NMR in Biomedicine vol 11 no 4-5 pp 157ndash1671998

[12] J S Suri S Singh and L Reden ldquoComputer vision andpattern recognition techniques for 2-D and 3-D MR cerebralcortical segmentation (Part I) a state-of-the-art reviewrdquo PatternAnalysis amp Applications vol 5 no 1 pp 46ndash76 2002

[13] A P Zijdenbos and B M Dawant ldquoBrain segmentation andwhite matter lesion detection in MR imagesrdquo Critical Reviewsin Biomedical Engineering vol 22 no 5-6 pp 401ndash465 1994

[14] K Price ldquoAnything you can do I can do better (No you canrsquot)rdquoComputer Vision Graphics and Image Processing vol 36 no 2-3pp 387ndash391 1986

[15] R de Boer H A Vrooman M A Ikram et al ldquoAccuracy andreproducibility study of automatic MRI brain tissue segmenta-tion methodsrdquo NeuroImage vol 51 no 3 pp 1047ndash1056 2010

[16] J de Bresser M P Portegies A Leemans G J Biessels L JKappelle andM A Viergever ldquoA comparison ofMR based seg-mentation methods for measuring brain atrophy progressionrdquoNeuroImage vol 54 no 2 pp 760ndash768 2011

Computational Intelligence and Neuroscience 15

[17] M B Cuadra L Cammoun T Butz O Cuisenaire and J-P Thiran ldquoComparison and validation of tissue modelizationand statistical classification methods in T1-weighted MR brainimagesrdquo IEEE Transactions on Medical Imaging vol 24 no 12pp 1548ndash1565 2005

[18] F Klauschen A Goldman V Barra A Meyer-Lindenbergand A Lundervold ldquoEvaluation of automated brain MR imagesegmentation and volumetrymethodsrdquoHuman BrainMappingvol 30 no 4 pp 1310ndash1327 2009

[19] H Zaidi T Ruest F Schoenahl and M-L Montandon ldquoCom-parative assessment of statistical brain MR image segmentationalgorithms and their impact on partial volume correction inPETrdquo NeuroImage vol 32 no 4 pp 1591ndash1607 2006

[20] J A Frazier S M Hodge J L Breeze et al ldquoDiagnostic and sexeffects on limbic volumes in early-onset bipolar disorder andschizophreniardquo Schizophrenia Bulletin vol 34 no 1 pp 37ndash462008

[21] A Klein and J Tourville ldquo101 labeled brain images and aconsistent human cortical labeling protocolrdquo Frontiers in Neu-roscience vol 6 article 171 2012

[22] B Van Ginneken T Heimann and M Styner ldquo3D segmen-tation in the clinic a grand challengerdquo in Proceedings of the10th International Conference on Medical Image Computingand Computer Assisted Intervention (MICCAI rsquo07) pp 7ndash15Brisbane Australia October-November 2007

[23] A M Dale B Fischl and M I Sereno ldquoCortical surface-based analysis I Segmentation and surface reconstructionrdquoNeuroImage vol 9 no 2 pp 179ndash194 1999

[24] B Fischl M I Sereno and A M Dale ldquoCortical surface-basedanalysis II inflation flattening and a surface-based coordinatesystemrdquo NeuroImage vol 9 no 2 pp 195ndash207 1999

[25] M Jenkinson C F Beckmann T E J BehrensMWWoolrichand S M Smith ldquoFSLrdquoNeuroImage vol 62 no 2 pp 782ndash7902012

[26] J Ashburner and K J Friston ldquoUnified segmentationrdquo Neu-roImage vol 26 no 3 pp 839ndash851 2005

[27] Y D Reijmer A Leemans M Brundel L J Kappelle and GJ Biessels ldquoDisruption of the cerebral white matter network isrelated to slowing of information processing speed in patientswith type 2 diabetesrdquoDiabetes vol 62 no 6 pp 2112ndash2115 2013

[28] S Klein M Staring K Murphy M A Viergever and J P WPluim ldquoElastix a toolbox for intensity-based medical imageregistrationrdquo IEEE Transactions onMedical Imaging vol 29 no1 pp 196ndash205 2010

[29] K J Friston J T Ashburner S J Kiebel T E Nichols and WD Penny Eds Statistical Parametric Mapping The Analysis ofFunctional Brain Images Academic Press 2011

[30] F Ritter T Boskamp A Homeyer et al ldquoMedical imageanalysis a visual approachrdquo IEEE Pulse vol 2 no 6 pp 60ndash702011

[31] L R Dice ldquoMeasures of the amount of ecologic associationbetween speciesrdquo Ecology vol 26 no 3 pp 297ndash302 1945

[32] D P Huttenlocher G A Klanderman and W J RucklidgeldquoComparing images using the Hausdorff distancerdquo IEEE Trans-actions on Pattern Analysis and Machine Intelligence vol 15 no9 pp 850ndash863 1993

[33] A Alansary A Soliman F Khalifa et al ldquoMAP-based frame-work for segmentation of MR brain images based on visualappearance and prior shaperdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal September 2013

[34] A Jog S Roy J L Prince and A Carass ldquoMR brain segmenta-tion using decision treesrdquo in Proceedings of the MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal Nagoya Japan September 2013

[35] R Katyal S Paneri and M Kuse ldquoGaussian intensity modelwith neighborhood cues for fluid-tissue categorization ofmulti-sequence MR brain imagesrdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal Nagoya Japan September 2013

[36] QMahmoodMAlipoor A Chodorowski AMehnert andMPersson ldquoMultimodal MR brain segmentation using Bayesian-based adaptive mean-shift (BAMS)rdquo in Proceedings of theMICCAI WorkshopsmdashThe MICCAI Grand Challenge on MRBrain Image Segmentation (MRBrainS rsquo13) The Midas JournalSeptember 2013

[37] A van Opbroek F van der Lijn andM de Bruijne ldquoAutomatedbrain-tissue segmentation by multi-feature SVM classificationrdquoin Proceedings of the MICCAI WorkshopsmdashThe MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal September 2013

[38] S Pereira J Festa J A Mariz N Sousa and C A SilvaldquoAutomatic brain tissue segmentation of multi-sequence MRimages using random decision forestsrdquo in Proceedings of theMICCAI Grand Challenge on MR Brain Image Segmentation(MRBrainS rsquo13) The Midas Journal Nagoya Japan September2013

[39] MRajchl J Baxter J Yuan T Peters andAKhan ldquoMulti-atlas-based segmentation with hierarchical max-flowrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[40] D Sarikaya L Zhao and J J Corso ldquoMulti-atlas brain MRIsegmentation with multiway cutrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[41] H Vrooman F van der Lijn and W Niessen ldquoAuto-kNNbrain tissue segmentation using automatically trained k-nearest-neighbor classificationrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[42] S Vyas P Burlina D Kleissas and R Mukherjee ldquoAutomatedwalks using machine learning for segmentationrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[43] C Wang and O Smedby ldquoFully automatic brain segmentationusing model-guided level sets and skeleton-based modelsrdquo inProceedings of theMICCAI Grand Challenge onMRBrain ImageSegmentation (MRBrainS rsquo13) The Midas Journal NagoyaJapan September 2013

[44] C Cortes and V Vapnik ldquoSupport-vector networksrdquo MachineLearning vol 20 no 3 pp 273ndash297 1995

[45] H A Vrooman C A Cocosco F van der Lijn et al ldquoMulti-spectral brain tissue segmentation using automatically trainedk-Nearest-Neighbor classificationrdquo NeuroImage vol 37 no 1pp 71ndash81 2007

[46] C Wang H Frimmel and O Smedby ldquoFast level-set basedimage segmentation using coherent propagationrdquo MedicalPhysics vol 41 no 7 Article ID 073501 2014

16 Computational Intelligence and Neuroscience

[47] A Jog S Roy A Carass and J L Prince ldquoMagnetic resonanceimage synthesis through patch regressionrdquo in Proceedings of theIEEE 10th International Symposium onBiomedical Imaging (ISBIrsquo13) pp 350ndash353 April 2013

[48] J G Sied A P Zijdenbos and A C Evans ldquoA non-parametricmethod for automatic correction of intensity non-uniformity inMRI datardquo IEEE Transactions on Medical Imaging vol 17 no 1pp 87ndash97 1998

[49] A Carass J Cuzzocreo M B Wheeler P-L Bazin S MResnick and J L Prince ldquoSimple paradigm for extra-cerebraltissue removal algorithm and analysisrdquoNeuroImage vol 56 no4 pp 1982ndash1992 2011

[50] M Modat G R Ridgway Z A Taylor et al ldquoFast free-form deformation using graphics processing unitsrdquo ComputerMethods and Programs in Biomedicine vol 98 no 3 pp 278ndash284 2010

[51] M Rajchl J Yuan J A White et al ldquoInteractive hierarchical-flow segmentation of scar tissue from late-enhancement cardiacmr imagesrdquo IEEE Transactions on Medical Imaging vol 33 no1 pp 159ndash172 2014

[52] QMahmood A Chodorowski AMehnert andM Persson ldquoAnovel Bayesian approach to adaptive mean shift segmentationof brain imagesrdquo in Proceedings of the 25th IEEE InternationalSymposiumonComputer-BasedMedical Systems (CBMS rsquo12) pp1ndash6 June 2012

[53] A El-Baz M Nitzken A Alansary A Soliman and MCasanova ldquoBrain Segmentation Method for Young Childrenand Adultsrdquo US Provisional Patent Application 13100

[54] A Alansary A Soliman M Nitzken et al ldquoAn integratedgeometrical and stochastic approach for accurate infant brainextractionrdquo in Proceedings of the IEEE International Conferenceon Image Processing (ICIP rsquo14) pp 3542ndash3546 October 2014

[55] V Popescu M Battaglini W S Hoogstrate et al ldquoOptimizingparameter choice for FSL-Brain Extraction Tool (BET) on 3DT1 images in multiple sclerosisrdquo NeuroImage vol 61 no 4 pp1484ndash1494 2012

[56] J L Whitwell W R Crum H C Watt and N C FoxldquoNormalization of cerebral volumes by use of intracranial vol-ume implications for longitudinal quantitative MR imagingrdquoAmerican Journal of Neuroradiology vol 22 no 8 pp 1483ndash1489 2001

[57] S M Smith ldquoFast robust automated brain extractionrdquo HumanBrain Mapping vol 17 no 3 pp 143ndash155 2002

[58] D W Shattuck G Prasad M Mirza K L Narr and A WToga ldquoOnline resource for validation of brain segmentationmethodsrdquo NeuroImage vol 45 no 2 pp 431ndash439 2009

[59] I Isgum M J N L Benders B Avants et al ldquoEvaluation ofautomatic neonatal brain segmentation algorithms the Neo-BrainS12 challengerdquo Medical Image Analysis vol 20 no 1 pp135ndash151 2015

[60] P Moeskops M J N L Benders S M Chita et al ldquoAutomaticsegmentation of MR brain images of preterm infants usingsupervised classificationrdquo NeuroImage 2015

[61] P Moeskops M A Viergever M J N L Benders and IIsgum ldquoEvaluation of an automatic brain segmentationmethoddeveloped for neonates on adult MR brain imagesrdquo in MedicalImaging Image Processing vol 9413 of Proceedings of SPIEInternational Society for Optics and Photonics Orlando FlaUSA February 2015

[62] L Wang Y Gao F Shi et al ldquoLINKS learning-based multi-source IntegratioN frameworK for Segmentation of infant brainimagesrdquo NeuroImage vol 108 pp 160ndash172 2015

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

10 Computational Intelligence and Neuroscience

2468

10121416

GM H-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(a)

2468

10121416

WMH-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(b)

5

10

15

20CSF H-95

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(c)

5

10

15

20H

-95

(mm

)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Brain (WM + GM) H-95

(d)

5

10

15

20

25

30

H-9

5 (m

m)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Intracranial volume (WM + GM + CSF) H-95

(e)

Figure 3 Boxplots presenting the evaluation results for the 95th-percentile Hausdorff distance (3) of the gray matter (GM) white matter(WM) cerebrospinal fluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all15 test datasets

Computational Intelligence and Neuroscience 11

0

10

20

30

GM AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(a)

0

10

20

30

40WM AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(b)

0

20

40

60

80

CSF AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(c)

0

5

10

15

20

25

30AV

D (

)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Brain (WM + GM) AVD

(d)

0

5

10

15

20

25

30

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Intracranial volume (WM + GM + CSF) AVD

(e)

Figure 4 Boxplots presenting the evaluation results for the absolute volume difference (4) of the gray matter (GM) white matter (WM)cerebrospinal fluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all 15 testdatasets

12 Computational Intelligence and Neuroscience

T2

UofL

N

FS

S2

‐FLAIR

BioImaging

Narsil

L‐Seg

2_QM

T1‐IR

CMIV

SPM12_T1

SPM12_T1_

LNMBr

_FLAIR

_IR_FLAIR

ains

T1

UB VPML Me

SPM12_T1_ I

FSL‐PVSeg

R

d B

R

F

eference

igr_neuro

MNAB

reeSurfer

BIG

Rob

SPM1

Jedi Mi

R2

arts

2_T1

nd Meld

Figure 5 Illustration of the segmentation results at the height of the basal ganglia (test subject 9 slice 22) The basal ganglia should besegmented as gray matter The first three images show the three MRI sequences The fourth image (reference) is the manually segmentedreference standard (yellow white matter light blue gray matter and dark blue cerebrospinal fluid) The results of the participatingsegmentation algorithms are ordered from the best overall performance (BIGR2) to the worst overall performance (LNMBrains) Majordifferences are present in the segmentation of the basal ganglia and the outer border of the cerebrospinal fluid The arrows indicate examplelocations where the segmentation results differ from the ground truth

Computational Intelligence and Neuroscience 13

T2

UofL BioIm

Nars

FSL‐S

S2_Q

‐FLAIR

aging (860)

il (892)

eg (453)

M (565)

T1‐IR

CMIV (90

SPM12_T1_F

SPM12_T1_IR

LNMBrains

5) UB

(831) SP

_F (815) F

(286)

T1

VPML Med (9

M12_T1_IR (60

SL‐PVSeg (525

R

10) Bigr_

5) MN

) FreeS

eference

neuro (896)

AB (889)

urfer (846)

BIGR2

Robarts

SPM12_T

Jedi Mind M

(852)

(810)

1 (683)

eld (769)

Figure 6 Illustration of the segmentation results in the presence of whitematter lesions (test subject 3 slice 31)Whitematter lesions (WMLs)should be labeled asWM the sensitivity (percentage ofWML voxels is labeled asWM) over all 15 datasets is presented between brackets afterthe team nameThe first three images show the threeMRI sequences T2-weighted fluid attenuated inversion recovery T1-weighted inversionrecovery and T1-weighted scanThe fourth image (reference) is themanually segmented reference standard (red whitematter lesions yellowwhite matter light blue gray matter and dark blue cerebrospinal fluid)The results of the segmentation algorithms are ordered from the bestoverall performance (BIGR2) to the worst overall performance (LNMBrains)The arrows indicate example locations where the segmentationresults differ from the ground truth

14 Computational Intelligence and Neuroscience

fromboth theMRBrainS and theNeoBrainS [59] (brain tissuesegmentation in neonates) challenge Two methods [60ndash62]specifically designed for neonatal brain tissue segmentationshowed a high performance for tissue segmentation on theMRBrainS data Applying algorithms to different types of datahas the potential to lead to new insights and more robustalgorithms The MRBrainS evaluation framework remainsopen for new contributions At the time of writing 21teams had submitted their results on the MRBrainS website(httpmrbrains13isiuunlresultsphp)

5 Conclusion

The MRBrainS challenge online evaluation framework pro-vides direct and objective comparison of automatic andsemiautomatic methods to segment GM WM CSF brainand ICV on 3T multisequence MRI data The first elevenparticipating methods are evaluated and presented in thispaper as well as three commonly used freeware packages(FreeSurfer FSL and SPM12) They provide a benchmarkfor future contributions to the framework The final rankingprovides a quick insight into the overall performance of theevaluated methods in comparison to each other whereasthe individual evaluation measures (Dice 95th-percentileHausdorff distance and absolute volume difference) percomponent (GM WM CSF brain and ICV) can aid inselecting the best method for a specific segmentation goal

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was financially supported by IMDI Grant104002002 (Brainbox) fromZonMw the Netherlands Organ-isation for Health Research and Development within kindsponsoring by Philips theUniversityMedical CenterUtrechtand Eindhoven University of TechnologyThe authors wouldlike to acknowledge the following members of the UtrechtVascular Cognitive Impairment Study Group who were notincluded as coauthors of this paper but were involved in therecruitment of study participants and MRI acquisition at theUMC Utrecht (in alphabetical order by department) E vanden Berg M Brundel S Heringa and L J Kappelle of theDepartment of Neurology P R Luijten and W P Th MMali of the Department of Radiology and A Algra and GE H M Rutten of the Julius Center for Health Sciences andPrimary CareThe research of Geert Jan Biessels and the VCIgroupwas financially supported byVIDIGrant 91711384 fromZonMw and by Grant 2010T073 of the Netherlands HeartFoundation The research of Jeroen de Bresser is financiallysupported by a research talent fellowship of the Univer-sity Medical Center Utrecht (Netherlands) The research ofAnnegreet vanOpbroek andMarleen de Bruijne is financiallysupported by a research grant from NWO (the NetherlandsOrganisation for Scientific Research) The authors would

like to acknowledge MeVis Medical Solutions AG (BremenGermany) for providing MeVisLab Duygu Sarikaya andLiang Zhao acknowledge their Advisor Professor JasonCorsofor his guidance Duygu Sarikaya is supported by NIH 1 R21CA160825-01 and Liang Zhao is partially supported by theChina Scholarship Council (CSC)

References

[1] I Driscoll C Davatzikos Y An et al ldquoLongitudinal pattern ofregional brain volume change differentiates normal aging fromMCIrdquo Neurology vol 72 no 22 pp 1906ndash1913 2009

[2] K Franke and C Gaser ldquoLongitudinal changes in individualBrainAGE in healthy aging mild cognitive impairment andAlzheimerrsquos diseaserdquoGeroPsych vol 25 no 4 pp 235ndash245 2012

[3] MA IkramH A VroomanMWVernooij et al ldquoBrain tissuevolumes in the general elderly populationThe Rotterdam ScanStudyrdquo Neurobiology of Aging vol 29 no 6 pp 882ndash890 2008

[4] A Giorgio and N de Stefano ldquoClinical use of brain volumetryrdquoJournal of Magnetic Resonance Imaging vol 37 no 1 pp 1ndash142013

[5] M W Vannier R L Butterfield D Jordan W A Murphy RG Levitt and M Gado ldquoMultispectral analysis of magneticresonance imagesrdquo Radiology vol 154 no 1 pp 221ndash224 1985

[6] M A Balafar A R Ramli M I Saripan and S MashohorldquoReview of brain MRI image segmentation methodsrdquo ArtificialIntelligence Review vol 33 no 3 pp 261ndash274 2010

[7] J C Bezdek L O Hall and L P Clarke ldquoReview of MR imagesegmentation techniques using pattern recognitionrdquo MedicalPhysics vol 20 no 4 pp 1033ndash1048 1993

[8] L P Clarke R P Velthuizen M A Camacho et al ldquoMRIsegmentation methods and applicationsrdquo Magnetic ResonanceImaging vol 13 no 3 pp 343ndash368 1995

[9] J S Duncan X Papademetris J Yang M Jackowski X Zengand L H Staib ldquoGeometric strategies for neuroanatomic ana-lysis from MRIrdquo NeuroImage vol 23 supplement 1 pp S34ndashS45 2004

[10] D L PhamC Xu and J L Prince ldquoCurrentmethods inmedicalimage segmentationrdquoAnnual Review of Biomedical Engineeringvol 2 no 2000 pp 315ndash337 2000

[11] N Saeed ldquoMagnetic resonance image segmentation usingpattern recognition and applied to image registration andquantitationrdquo NMR in Biomedicine vol 11 no 4-5 pp 157ndash1671998

[12] J S Suri S Singh and L Reden ldquoComputer vision andpattern recognition techniques for 2-D and 3-D MR cerebralcortical segmentation (Part I) a state-of-the-art reviewrdquo PatternAnalysis amp Applications vol 5 no 1 pp 46ndash76 2002

[13] A P Zijdenbos and B M Dawant ldquoBrain segmentation andwhite matter lesion detection in MR imagesrdquo Critical Reviewsin Biomedical Engineering vol 22 no 5-6 pp 401ndash465 1994

[14] K Price ldquoAnything you can do I can do better (No you canrsquot)rdquoComputer Vision Graphics and Image Processing vol 36 no 2-3pp 387ndash391 1986

[15] R de Boer H A Vrooman M A Ikram et al ldquoAccuracy andreproducibility study of automatic MRI brain tissue segmenta-tion methodsrdquo NeuroImage vol 51 no 3 pp 1047ndash1056 2010

[16] J de Bresser M P Portegies A Leemans G J Biessels L JKappelle andM A Viergever ldquoA comparison ofMR based seg-mentation methods for measuring brain atrophy progressionrdquoNeuroImage vol 54 no 2 pp 760ndash768 2011

Computational Intelligence and Neuroscience 15

[17] M B Cuadra L Cammoun T Butz O Cuisenaire and J-P Thiran ldquoComparison and validation of tissue modelizationand statistical classification methods in T1-weighted MR brainimagesrdquo IEEE Transactions on Medical Imaging vol 24 no 12pp 1548ndash1565 2005

[18] F Klauschen A Goldman V Barra A Meyer-Lindenbergand A Lundervold ldquoEvaluation of automated brain MR imagesegmentation and volumetrymethodsrdquoHuman BrainMappingvol 30 no 4 pp 1310ndash1327 2009

[19] H Zaidi T Ruest F Schoenahl and M-L Montandon ldquoCom-parative assessment of statistical brain MR image segmentationalgorithms and their impact on partial volume correction inPETrdquo NeuroImage vol 32 no 4 pp 1591ndash1607 2006

[20] J A Frazier S M Hodge J L Breeze et al ldquoDiagnostic and sexeffects on limbic volumes in early-onset bipolar disorder andschizophreniardquo Schizophrenia Bulletin vol 34 no 1 pp 37ndash462008

[21] A Klein and J Tourville ldquo101 labeled brain images and aconsistent human cortical labeling protocolrdquo Frontiers in Neu-roscience vol 6 article 171 2012

[22] B Van Ginneken T Heimann and M Styner ldquo3D segmen-tation in the clinic a grand challengerdquo in Proceedings of the10th International Conference on Medical Image Computingand Computer Assisted Intervention (MICCAI rsquo07) pp 7ndash15Brisbane Australia October-November 2007

[23] A M Dale B Fischl and M I Sereno ldquoCortical surface-based analysis I Segmentation and surface reconstructionrdquoNeuroImage vol 9 no 2 pp 179ndash194 1999

[24] B Fischl M I Sereno and A M Dale ldquoCortical surface-basedanalysis II inflation flattening and a surface-based coordinatesystemrdquo NeuroImage vol 9 no 2 pp 195ndash207 1999

[25] M Jenkinson C F Beckmann T E J BehrensMWWoolrichand S M Smith ldquoFSLrdquoNeuroImage vol 62 no 2 pp 782ndash7902012

[26] J Ashburner and K J Friston ldquoUnified segmentationrdquo Neu-roImage vol 26 no 3 pp 839ndash851 2005

[27] Y D Reijmer A Leemans M Brundel L J Kappelle and GJ Biessels ldquoDisruption of the cerebral white matter network isrelated to slowing of information processing speed in patientswith type 2 diabetesrdquoDiabetes vol 62 no 6 pp 2112ndash2115 2013

[28] S Klein M Staring K Murphy M A Viergever and J P WPluim ldquoElastix a toolbox for intensity-based medical imageregistrationrdquo IEEE Transactions onMedical Imaging vol 29 no1 pp 196ndash205 2010

[29] K J Friston J T Ashburner S J Kiebel T E Nichols and WD Penny Eds Statistical Parametric Mapping The Analysis ofFunctional Brain Images Academic Press 2011

[30] F Ritter T Boskamp A Homeyer et al ldquoMedical imageanalysis a visual approachrdquo IEEE Pulse vol 2 no 6 pp 60ndash702011

[31] L R Dice ldquoMeasures of the amount of ecologic associationbetween speciesrdquo Ecology vol 26 no 3 pp 297ndash302 1945

[32] D P Huttenlocher G A Klanderman and W J RucklidgeldquoComparing images using the Hausdorff distancerdquo IEEE Trans-actions on Pattern Analysis and Machine Intelligence vol 15 no9 pp 850ndash863 1993

[33] A Alansary A Soliman F Khalifa et al ldquoMAP-based frame-work for segmentation of MR brain images based on visualappearance and prior shaperdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal September 2013

[34] A Jog S Roy J L Prince and A Carass ldquoMR brain segmenta-tion using decision treesrdquo in Proceedings of the MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal Nagoya Japan September 2013

[35] R Katyal S Paneri and M Kuse ldquoGaussian intensity modelwith neighborhood cues for fluid-tissue categorization ofmulti-sequence MR brain imagesrdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal Nagoya Japan September 2013

[36] QMahmoodMAlipoor A Chodorowski AMehnert andMPersson ldquoMultimodal MR brain segmentation using Bayesian-based adaptive mean-shift (BAMS)rdquo in Proceedings of theMICCAI WorkshopsmdashThe MICCAI Grand Challenge on MRBrain Image Segmentation (MRBrainS rsquo13) The Midas JournalSeptember 2013

[37] A van Opbroek F van der Lijn andM de Bruijne ldquoAutomatedbrain-tissue segmentation by multi-feature SVM classificationrdquoin Proceedings of the MICCAI WorkshopsmdashThe MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal September 2013

[38] S Pereira J Festa J A Mariz N Sousa and C A SilvaldquoAutomatic brain tissue segmentation of multi-sequence MRimages using random decision forestsrdquo in Proceedings of theMICCAI Grand Challenge on MR Brain Image Segmentation(MRBrainS rsquo13) The Midas Journal Nagoya Japan September2013

[39] MRajchl J Baxter J Yuan T Peters andAKhan ldquoMulti-atlas-based segmentation with hierarchical max-flowrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[40] D Sarikaya L Zhao and J J Corso ldquoMulti-atlas brain MRIsegmentation with multiway cutrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[41] H Vrooman F van der Lijn and W Niessen ldquoAuto-kNNbrain tissue segmentation using automatically trained k-nearest-neighbor classificationrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[42] S Vyas P Burlina D Kleissas and R Mukherjee ldquoAutomatedwalks using machine learning for segmentationrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[43] C Wang and O Smedby ldquoFully automatic brain segmentationusing model-guided level sets and skeleton-based modelsrdquo inProceedings of theMICCAI Grand Challenge onMRBrain ImageSegmentation (MRBrainS rsquo13) The Midas Journal NagoyaJapan September 2013

[44] C Cortes and V Vapnik ldquoSupport-vector networksrdquo MachineLearning vol 20 no 3 pp 273ndash297 1995

[45] H A Vrooman C A Cocosco F van der Lijn et al ldquoMulti-spectral brain tissue segmentation using automatically trainedk-Nearest-Neighbor classificationrdquo NeuroImage vol 37 no 1pp 71ndash81 2007

[46] C Wang H Frimmel and O Smedby ldquoFast level-set basedimage segmentation using coherent propagationrdquo MedicalPhysics vol 41 no 7 Article ID 073501 2014

16 Computational Intelligence and Neuroscience

[47] A Jog S Roy A Carass and J L Prince ldquoMagnetic resonanceimage synthesis through patch regressionrdquo in Proceedings of theIEEE 10th International Symposium onBiomedical Imaging (ISBIrsquo13) pp 350ndash353 April 2013

[48] J G Sied A P Zijdenbos and A C Evans ldquoA non-parametricmethod for automatic correction of intensity non-uniformity inMRI datardquo IEEE Transactions on Medical Imaging vol 17 no 1pp 87ndash97 1998

[49] A Carass J Cuzzocreo M B Wheeler P-L Bazin S MResnick and J L Prince ldquoSimple paradigm for extra-cerebraltissue removal algorithm and analysisrdquoNeuroImage vol 56 no4 pp 1982ndash1992 2011

[50] M Modat G R Ridgway Z A Taylor et al ldquoFast free-form deformation using graphics processing unitsrdquo ComputerMethods and Programs in Biomedicine vol 98 no 3 pp 278ndash284 2010

[51] M Rajchl J Yuan J A White et al ldquoInteractive hierarchical-flow segmentation of scar tissue from late-enhancement cardiacmr imagesrdquo IEEE Transactions on Medical Imaging vol 33 no1 pp 159ndash172 2014

[52] QMahmood A Chodorowski AMehnert andM Persson ldquoAnovel Bayesian approach to adaptive mean shift segmentationof brain imagesrdquo in Proceedings of the 25th IEEE InternationalSymposiumonComputer-BasedMedical Systems (CBMS rsquo12) pp1ndash6 June 2012

[53] A El-Baz M Nitzken A Alansary A Soliman and MCasanova ldquoBrain Segmentation Method for Young Childrenand Adultsrdquo US Provisional Patent Application 13100

[54] A Alansary A Soliman M Nitzken et al ldquoAn integratedgeometrical and stochastic approach for accurate infant brainextractionrdquo in Proceedings of the IEEE International Conferenceon Image Processing (ICIP rsquo14) pp 3542ndash3546 October 2014

[55] V Popescu M Battaglini W S Hoogstrate et al ldquoOptimizingparameter choice for FSL-Brain Extraction Tool (BET) on 3DT1 images in multiple sclerosisrdquo NeuroImage vol 61 no 4 pp1484ndash1494 2012

[56] J L Whitwell W R Crum H C Watt and N C FoxldquoNormalization of cerebral volumes by use of intracranial vol-ume implications for longitudinal quantitative MR imagingrdquoAmerican Journal of Neuroradiology vol 22 no 8 pp 1483ndash1489 2001

[57] S M Smith ldquoFast robust automated brain extractionrdquo HumanBrain Mapping vol 17 no 3 pp 143ndash155 2002

[58] D W Shattuck G Prasad M Mirza K L Narr and A WToga ldquoOnline resource for validation of brain segmentationmethodsrdquo NeuroImage vol 45 no 2 pp 431ndash439 2009

[59] I Isgum M J N L Benders B Avants et al ldquoEvaluation ofautomatic neonatal brain segmentation algorithms the Neo-BrainS12 challengerdquo Medical Image Analysis vol 20 no 1 pp135ndash151 2015

[60] P Moeskops M J N L Benders S M Chita et al ldquoAutomaticsegmentation of MR brain images of preterm infants usingsupervised classificationrdquo NeuroImage 2015

[61] P Moeskops M A Viergever M J N L Benders and IIsgum ldquoEvaluation of an automatic brain segmentationmethoddeveloped for neonates on adult MR brain imagesrdquo in MedicalImaging Image Processing vol 9413 of Proceedings of SPIEInternational Society for Optics and Photonics Orlando FlaUSA February 2015

[62] L Wang Y Gao F Shi et al ldquoLINKS learning-based multi-source IntegratioN frameworK for Segmentation of infant brainimagesrdquo NeuroImage vol 108 pp 160ndash172 2015

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience 11

0

10

20

30

GM AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(a)

0

10

20

30

40WM AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(b)

0

20

40

60

80

CSF AVD

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

(c)

0

5

10

15

20

25

30AV

D (

)

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Brain (WM + GM) AVD

(d)

0

5

10

15

20

25

30

AVD

()

BIG

R2U

ofL

BioI

mag

ing

CMIV

UB

VPM

L M

edBi

gr_n

euro

Roba

rts

Nar

silSP

M12

_T1_

FLA

IRSP

M12

_T1_

IRM

NA

BSP

M12

_T1

FSL-

Seg

SPM

12_T

1_IR

_FLA

IRFS

L-PV

Seg

Free

Surfe

rJe

di M

ind

Meld

S2_Q

MLN

MBr

ains

Intracranial volume (WM + GM + CSF) AVD

(e)

Figure 4 Boxplots presenting the evaluation results for the absolute volume difference (4) of the gray matter (GM) white matter (WM)cerebrospinal fluid (CSF) and brain and ICV segmentations for each of the participating algorithms and freeware packages over all 15 testdatasets

12 Computational Intelligence and Neuroscience

T2

UofL

N

FS

S2

‐FLAIR

BioImaging

Narsil

L‐Seg

2_QM

T1‐IR

CMIV

SPM12_T1

SPM12_T1_

LNMBr

_FLAIR

_IR_FLAIR

ains

T1

UB VPML Me

SPM12_T1_ I

FSL‐PVSeg

R

d B

R

F

eference

igr_neuro

MNAB

reeSurfer

BIG

Rob

SPM1

Jedi Mi

R2

arts

2_T1

nd Meld

Figure 5 Illustration of the segmentation results at the height of the basal ganglia (test subject 9 slice 22) The basal ganglia should besegmented as gray matter The first three images show the three MRI sequences The fourth image (reference) is the manually segmentedreference standard (yellow white matter light blue gray matter and dark blue cerebrospinal fluid) The results of the participatingsegmentation algorithms are ordered from the best overall performance (BIGR2) to the worst overall performance (LNMBrains) Majordifferences are present in the segmentation of the basal ganglia and the outer border of the cerebrospinal fluid The arrows indicate examplelocations where the segmentation results differ from the ground truth

Computational Intelligence and Neuroscience 13

T2

UofL BioIm

Nars

FSL‐S

S2_Q

‐FLAIR

aging (860)

il (892)

eg (453)

M (565)

T1‐IR

CMIV (90

SPM12_T1_F

SPM12_T1_IR

LNMBrains

5) UB

(831) SP

_F (815) F

(286)

T1

VPML Med (9

M12_T1_IR (60

SL‐PVSeg (525

R

10) Bigr_

5) MN

) FreeS

eference

neuro (896)

AB (889)

urfer (846)

BIGR2

Robarts

SPM12_T

Jedi Mind M

(852)

(810)

1 (683)

eld (769)

Figure 6 Illustration of the segmentation results in the presence of whitematter lesions (test subject 3 slice 31)Whitematter lesions (WMLs)should be labeled asWM the sensitivity (percentage ofWML voxels is labeled asWM) over all 15 datasets is presented between brackets afterthe team nameThe first three images show the threeMRI sequences T2-weighted fluid attenuated inversion recovery T1-weighted inversionrecovery and T1-weighted scanThe fourth image (reference) is themanually segmented reference standard (red whitematter lesions yellowwhite matter light blue gray matter and dark blue cerebrospinal fluid)The results of the segmentation algorithms are ordered from the bestoverall performance (BIGR2) to the worst overall performance (LNMBrains)The arrows indicate example locations where the segmentationresults differ from the ground truth

14 Computational Intelligence and Neuroscience

fromboth theMRBrainS and theNeoBrainS [59] (brain tissuesegmentation in neonates) challenge Two methods [60ndash62]specifically designed for neonatal brain tissue segmentationshowed a high performance for tissue segmentation on theMRBrainS data Applying algorithms to different types of datahas the potential to lead to new insights and more robustalgorithms The MRBrainS evaluation framework remainsopen for new contributions At the time of writing 21teams had submitted their results on the MRBrainS website(httpmrbrains13isiuunlresultsphp)

5 Conclusion

The MRBrainS challenge online evaluation framework pro-vides direct and objective comparison of automatic andsemiautomatic methods to segment GM WM CSF brainand ICV on 3T multisequence MRI data The first elevenparticipating methods are evaluated and presented in thispaper as well as three commonly used freeware packages(FreeSurfer FSL and SPM12) They provide a benchmarkfor future contributions to the framework The final rankingprovides a quick insight into the overall performance of theevaluated methods in comparison to each other whereasthe individual evaluation measures (Dice 95th-percentileHausdorff distance and absolute volume difference) percomponent (GM WM CSF brain and ICV) can aid inselecting the best method for a specific segmentation goal

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was financially supported by IMDI Grant104002002 (Brainbox) fromZonMw the Netherlands Organ-isation for Health Research and Development within kindsponsoring by Philips theUniversityMedical CenterUtrechtand Eindhoven University of TechnologyThe authors wouldlike to acknowledge the following members of the UtrechtVascular Cognitive Impairment Study Group who were notincluded as coauthors of this paper but were involved in therecruitment of study participants and MRI acquisition at theUMC Utrecht (in alphabetical order by department) E vanden Berg M Brundel S Heringa and L J Kappelle of theDepartment of Neurology P R Luijten and W P Th MMali of the Department of Radiology and A Algra and GE H M Rutten of the Julius Center for Health Sciences andPrimary CareThe research of Geert Jan Biessels and the VCIgroupwas financially supported byVIDIGrant 91711384 fromZonMw and by Grant 2010T073 of the Netherlands HeartFoundation The research of Jeroen de Bresser is financiallysupported by a research talent fellowship of the Univer-sity Medical Center Utrecht (Netherlands) The research ofAnnegreet vanOpbroek andMarleen de Bruijne is financiallysupported by a research grant from NWO (the NetherlandsOrganisation for Scientific Research) The authors would

like to acknowledge MeVis Medical Solutions AG (BremenGermany) for providing MeVisLab Duygu Sarikaya andLiang Zhao acknowledge their Advisor Professor JasonCorsofor his guidance Duygu Sarikaya is supported by NIH 1 R21CA160825-01 and Liang Zhao is partially supported by theChina Scholarship Council (CSC)

References

[1] I Driscoll C Davatzikos Y An et al ldquoLongitudinal pattern ofregional brain volume change differentiates normal aging fromMCIrdquo Neurology vol 72 no 22 pp 1906ndash1913 2009

[2] K Franke and C Gaser ldquoLongitudinal changes in individualBrainAGE in healthy aging mild cognitive impairment andAlzheimerrsquos diseaserdquoGeroPsych vol 25 no 4 pp 235ndash245 2012

[3] MA IkramH A VroomanMWVernooij et al ldquoBrain tissuevolumes in the general elderly populationThe Rotterdam ScanStudyrdquo Neurobiology of Aging vol 29 no 6 pp 882ndash890 2008

[4] A Giorgio and N de Stefano ldquoClinical use of brain volumetryrdquoJournal of Magnetic Resonance Imaging vol 37 no 1 pp 1ndash142013

[5] M W Vannier R L Butterfield D Jordan W A Murphy RG Levitt and M Gado ldquoMultispectral analysis of magneticresonance imagesrdquo Radiology vol 154 no 1 pp 221ndash224 1985

[6] M A Balafar A R Ramli M I Saripan and S MashohorldquoReview of brain MRI image segmentation methodsrdquo ArtificialIntelligence Review vol 33 no 3 pp 261ndash274 2010

[7] J C Bezdek L O Hall and L P Clarke ldquoReview of MR imagesegmentation techniques using pattern recognitionrdquo MedicalPhysics vol 20 no 4 pp 1033ndash1048 1993

[8] L P Clarke R P Velthuizen M A Camacho et al ldquoMRIsegmentation methods and applicationsrdquo Magnetic ResonanceImaging vol 13 no 3 pp 343ndash368 1995

[9] J S Duncan X Papademetris J Yang M Jackowski X Zengand L H Staib ldquoGeometric strategies for neuroanatomic ana-lysis from MRIrdquo NeuroImage vol 23 supplement 1 pp S34ndashS45 2004

[10] D L PhamC Xu and J L Prince ldquoCurrentmethods inmedicalimage segmentationrdquoAnnual Review of Biomedical Engineeringvol 2 no 2000 pp 315ndash337 2000

[11] N Saeed ldquoMagnetic resonance image segmentation usingpattern recognition and applied to image registration andquantitationrdquo NMR in Biomedicine vol 11 no 4-5 pp 157ndash1671998

[12] J S Suri S Singh and L Reden ldquoComputer vision andpattern recognition techniques for 2-D and 3-D MR cerebralcortical segmentation (Part I) a state-of-the-art reviewrdquo PatternAnalysis amp Applications vol 5 no 1 pp 46ndash76 2002

[13] A P Zijdenbos and B M Dawant ldquoBrain segmentation andwhite matter lesion detection in MR imagesrdquo Critical Reviewsin Biomedical Engineering vol 22 no 5-6 pp 401ndash465 1994

[14] K Price ldquoAnything you can do I can do better (No you canrsquot)rdquoComputer Vision Graphics and Image Processing vol 36 no 2-3pp 387ndash391 1986

[15] R de Boer H A Vrooman M A Ikram et al ldquoAccuracy andreproducibility study of automatic MRI brain tissue segmenta-tion methodsrdquo NeuroImage vol 51 no 3 pp 1047ndash1056 2010

[16] J de Bresser M P Portegies A Leemans G J Biessels L JKappelle andM A Viergever ldquoA comparison ofMR based seg-mentation methods for measuring brain atrophy progressionrdquoNeuroImage vol 54 no 2 pp 760ndash768 2011

Computational Intelligence and Neuroscience 15

[17] M B Cuadra L Cammoun T Butz O Cuisenaire and J-P Thiran ldquoComparison and validation of tissue modelizationand statistical classification methods in T1-weighted MR brainimagesrdquo IEEE Transactions on Medical Imaging vol 24 no 12pp 1548ndash1565 2005

[18] F Klauschen A Goldman V Barra A Meyer-Lindenbergand A Lundervold ldquoEvaluation of automated brain MR imagesegmentation and volumetrymethodsrdquoHuman BrainMappingvol 30 no 4 pp 1310ndash1327 2009

[19] H Zaidi T Ruest F Schoenahl and M-L Montandon ldquoCom-parative assessment of statistical brain MR image segmentationalgorithms and their impact on partial volume correction inPETrdquo NeuroImage vol 32 no 4 pp 1591ndash1607 2006

[20] J A Frazier S M Hodge J L Breeze et al ldquoDiagnostic and sexeffects on limbic volumes in early-onset bipolar disorder andschizophreniardquo Schizophrenia Bulletin vol 34 no 1 pp 37ndash462008

[21] A Klein and J Tourville ldquo101 labeled brain images and aconsistent human cortical labeling protocolrdquo Frontiers in Neu-roscience vol 6 article 171 2012

[22] B Van Ginneken T Heimann and M Styner ldquo3D segmen-tation in the clinic a grand challengerdquo in Proceedings of the10th International Conference on Medical Image Computingand Computer Assisted Intervention (MICCAI rsquo07) pp 7ndash15Brisbane Australia October-November 2007

[23] A M Dale B Fischl and M I Sereno ldquoCortical surface-based analysis I Segmentation and surface reconstructionrdquoNeuroImage vol 9 no 2 pp 179ndash194 1999

[24] B Fischl M I Sereno and A M Dale ldquoCortical surface-basedanalysis II inflation flattening and a surface-based coordinatesystemrdquo NeuroImage vol 9 no 2 pp 195ndash207 1999

[25] M Jenkinson C F Beckmann T E J BehrensMWWoolrichand S M Smith ldquoFSLrdquoNeuroImage vol 62 no 2 pp 782ndash7902012

[26] J Ashburner and K J Friston ldquoUnified segmentationrdquo Neu-roImage vol 26 no 3 pp 839ndash851 2005

[27] Y D Reijmer A Leemans M Brundel L J Kappelle and GJ Biessels ldquoDisruption of the cerebral white matter network isrelated to slowing of information processing speed in patientswith type 2 diabetesrdquoDiabetes vol 62 no 6 pp 2112ndash2115 2013

[28] S Klein M Staring K Murphy M A Viergever and J P WPluim ldquoElastix a toolbox for intensity-based medical imageregistrationrdquo IEEE Transactions onMedical Imaging vol 29 no1 pp 196ndash205 2010

[29] K J Friston J T Ashburner S J Kiebel T E Nichols and WD Penny Eds Statistical Parametric Mapping The Analysis ofFunctional Brain Images Academic Press 2011

[30] F Ritter T Boskamp A Homeyer et al ldquoMedical imageanalysis a visual approachrdquo IEEE Pulse vol 2 no 6 pp 60ndash702011

[31] L R Dice ldquoMeasures of the amount of ecologic associationbetween speciesrdquo Ecology vol 26 no 3 pp 297ndash302 1945

[32] D P Huttenlocher G A Klanderman and W J RucklidgeldquoComparing images using the Hausdorff distancerdquo IEEE Trans-actions on Pattern Analysis and Machine Intelligence vol 15 no9 pp 850ndash863 1993

[33] A Alansary A Soliman F Khalifa et al ldquoMAP-based frame-work for segmentation of MR brain images based on visualappearance and prior shaperdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal September 2013

[34] A Jog S Roy J L Prince and A Carass ldquoMR brain segmenta-tion using decision treesrdquo in Proceedings of the MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal Nagoya Japan September 2013

[35] R Katyal S Paneri and M Kuse ldquoGaussian intensity modelwith neighborhood cues for fluid-tissue categorization ofmulti-sequence MR brain imagesrdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal Nagoya Japan September 2013

[36] QMahmoodMAlipoor A Chodorowski AMehnert andMPersson ldquoMultimodal MR brain segmentation using Bayesian-based adaptive mean-shift (BAMS)rdquo in Proceedings of theMICCAI WorkshopsmdashThe MICCAI Grand Challenge on MRBrain Image Segmentation (MRBrainS rsquo13) The Midas JournalSeptember 2013

[37] A van Opbroek F van der Lijn andM de Bruijne ldquoAutomatedbrain-tissue segmentation by multi-feature SVM classificationrdquoin Proceedings of the MICCAI WorkshopsmdashThe MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal September 2013

[38] S Pereira J Festa J A Mariz N Sousa and C A SilvaldquoAutomatic brain tissue segmentation of multi-sequence MRimages using random decision forestsrdquo in Proceedings of theMICCAI Grand Challenge on MR Brain Image Segmentation(MRBrainS rsquo13) The Midas Journal Nagoya Japan September2013

[39] MRajchl J Baxter J Yuan T Peters andAKhan ldquoMulti-atlas-based segmentation with hierarchical max-flowrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[40] D Sarikaya L Zhao and J J Corso ldquoMulti-atlas brain MRIsegmentation with multiway cutrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[41] H Vrooman F van der Lijn and W Niessen ldquoAuto-kNNbrain tissue segmentation using automatically trained k-nearest-neighbor classificationrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[42] S Vyas P Burlina D Kleissas and R Mukherjee ldquoAutomatedwalks using machine learning for segmentationrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[43] C Wang and O Smedby ldquoFully automatic brain segmentationusing model-guided level sets and skeleton-based modelsrdquo inProceedings of theMICCAI Grand Challenge onMRBrain ImageSegmentation (MRBrainS rsquo13) The Midas Journal NagoyaJapan September 2013

[44] C Cortes and V Vapnik ldquoSupport-vector networksrdquo MachineLearning vol 20 no 3 pp 273ndash297 1995

[45] H A Vrooman C A Cocosco F van der Lijn et al ldquoMulti-spectral brain tissue segmentation using automatically trainedk-Nearest-Neighbor classificationrdquo NeuroImage vol 37 no 1pp 71ndash81 2007

[46] C Wang H Frimmel and O Smedby ldquoFast level-set basedimage segmentation using coherent propagationrdquo MedicalPhysics vol 41 no 7 Article ID 073501 2014

16 Computational Intelligence and Neuroscience

[47] A Jog S Roy A Carass and J L Prince ldquoMagnetic resonanceimage synthesis through patch regressionrdquo in Proceedings of theIEEE 10th International Symposium onBiomedical Imaging (ISBIrsquo13) pp 350ndash353 April 2013

[48] J G Sied A P Zijdenbos and A C Evans ldquoA non-parametricmethod for automatic correction of intensity non-uniformity inMRI datardquo IEEE Transactions on Medical Imaging vol 17 no 1pp 87ndash97 1998

[49] A Carass J Cuzzocreo M B Wheeler P-L Bazin S MResnick and J L Prince ldquoSimple paradigm for extra-cerebraltissue removal algorithm and analysisrdquoNeuroImage vol 56 no4 pp 1982ndash1992 2011

[50] M Modat G R Ridgway Z A Taylor et al ldquoFast free-form deformation using graphics processing unitsrdquo ComputerMethods and Programs in Biomedicine vol 98 no 3 pp 278ndash284 2010

[51] M Rajchl J Yuan J A White et al ldquoInteractive hierarchical-flow segmentation of scar tissue from late-enhancement cardiacmr imagesrdquo IEEE Transactions on Medical Imaging vol 33 no1 pp 159ndash172 2014

[52] QMahmood A Chodorowski AMehnert andM Persson ldquoAnovel Bayesian approach to adaptive mean shift segmentationof brain imagesrdquo in Proceedings of the 25th IEEE InternationalSymposiumonComputer-BasedMedical Systems (CBMS rsquo12) pp1ndash6 June 2012

[53] A El-Baz M Nitzken A Alansary A Soliman and MCasanova ldquoBrain Segmentation Method for Young Childrenand Adultsrdquo US Provisional Patent Application 13100

[54] A Alansary A Soliman M Nitzken et al ldquoAn integratedgeometrical and stochastic approach for accurate infant brainextractionrdquo in Proceedings of the IEEE International Conferenceon Image Processing (ICIP rsquo14) pp 3542ndash3546 October 2014

[55] V Popescu M Battaglini W S Hoogstrate et al ldquoOptimizingparameter choice for FSL-Brain Extraction Tool (BET) on 3DT1 images in multiple sclerosisrdquo NeuroImage vol 61 no 4 pp1484ndash1494 2012

[56] J L Whitwell W R Crum H C Watt and N C FoxldquoNormalization of cerebral volumes by use of intracranial vol-ume implications for longitudinal quantitative MR imagingrdquoAmerican Journal of Neuroradiology vol 22 no 8 pp 1483ndash1489 2001

[57] S M Smith ldquoFast robust automated brain extractionrdquo HumanBrain Mapping vol 17 no 3 pp 143ndash155 2002

[58] D W Shattuck G Prasad M Mirza K L Narr and A WToga ldquoOnline resource for validation of brain segmentationmethodsrdquo NeuroImage vol 45 no 2 pp 431ndash439 2009

[59] I Isgum M J N L Benders B Avants et al ldquoEvaluation ofautomatic neonatal brain segmentation algorithms the Neo-BrainS12 challengerdquo Medical Image Analysis vol 20 no 1 pp135ndash151 2015

[60] P Moeskops M J N L Benders S M Chita et al ldquoAutomaticsegmentation of MR brain images of preterm infants usingsupervised classificationrdquo NeuroImage 2015

[61] P Moeskops M A Viergever M J N L Benders and IIsgum ldquoEvaluation of an automatic brain segmentationmethoddeveloped for neonates on adult MR brain imagesrdquo in MedicalImaging Image Processing vol 9413 of Proceedings of SPIEInternational Society for Optics and Photonics Orlando FlaUSA February 2015

[62] L Wang Y Gao F Shi et al ldquoLINKS learning-based multi-source IntegratioN frameworK for Segmentation of infant brainimagesrdquo NeuroImage vol 108 pp 160ndash172 2015

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

12 Computational Intelligence and Neuroscience

T2

UofL

N

FS

S2

‐FLAIR

BioImaging

Narsil

L‐Seg

2_QM

T1‐IR

CMIV

SPM12_T1

SPM12_T1_

LNMBr

_FLAIR

_IR_FLAIR

ains

T1

UB VPML Me

SPM12_T1_ I

FSL‐PVSeg

R

d B

R

F

eference

igr_neuro

MNAB

reeSurfer

BIG

Rob

SPM1

Jedi Mi

R2

arts

2_T1

nd Meld

Figure 5 Illustration of the segmentation results at the height of the basal ganglia (test subject 9 slice 22) The basal ganglia should besegmented as gray matter The first three images show the three MRI sequences The fourth image (reference) is the manually segmentedreference standard (yellow white matter light blue gray matter and dark blue cerebrospinal fluid) The results of the participatingsegmentation algorithms are ordered from the best overall performance (BIGR2) to the worst overall performance (LNMBrains) Majordifferences are present in the segmentation of the basal ganglia and the outer border of the cerebrospinal fluid The arrows indicate examplelocations where the segmentation results differ from the ground truth

Computational Intelligence and Neuroscience 13

T2

UofL BioIm

Nars

FSL‐S

S2_Q

‐FLAIR

aging (860)

il (892)

eg (453)

M (565)

T1‐IR

CMIV (90

SPM12_T1_F

SPM12_T1_IR

LNMBrains

5) UB

(831) SP

_F (815) F

(286)

T1

VPML Med (9

M12_T1_IR (60

SL‐PVSeg (525

R

10) Bigr_

5) MN

) FreeS

eference

neuro (896)

AB (889)

urfer (846)

BIGR2

Robarts

SPM12_T

Jedi Mind M

(852)

(810)

1 (683)

eld (769)

Figure 6 Illustration of the segmentation results in the presence of whitematter lesions (test subject 3 slice 31)Whitematter lesions (WMLs)should be labeled asWM the sensitivity (percentage ofWML voxels is labeled asWM) over all 15 datasets is presented between brackets afterthe team nameThe first three images show the threeMRI sequences T2-weighted fluid attenuated inversion recovery T1-weighted inversionrecovery and T1-weighted scanThe fourth image (reference) is themanually segmented reference standard (red whitematter lesions yellowwhite matter light blue gray matter and dark blue cerebrospinal fluid)The results of the segmentation algorithms are ordered from the bestoverall performance (BIGR2) to the worst overall performance (LNMBrains)The arrows indicate example locations where the segmentationresults differ from the ground truth

14 Computational Intelligence and Neuroscience

fromboth theMRBrainS and theNeoBrainS [59] (brain tissuesegmentation in neonates) challenge Two methods [60ndash62]specifically designed for neonatal brain tissue segmentationshowed a high performance for tissue segmentation on theMRBrainS data Applying algorithms to different types of datahas the potential to lead to new insights and more robustalgorithms The MRBrainS evaluation framework remainsopen for new contributions At the time of writing 21teams had submitted their results on the MRBrainS website(httpmrbrains13isiuunlresultsphp)

5 Conclusion

The MRBrainS challenge online evaluation framework pro-vides direct and objective comparison of automatic andsemiautomatic methods to segment GM WM CSF brainand ICV on 3T multisequence MRI data The first elevenparticipating methods are evaluated and presented in thispaper as well as three commonly used freeware packages(FreeSurfer FSL and SPM12) They provide a benchmarkfor future contributions to the framework The final rankingprovides a quick insight into the overall performance of theevaluated methods in comparison to each other whereasthe individual evaluation measures (Dice 95th-percentileHausdorff distance and absolute volume difference) percomponent (GM WM CSF brain and ICV) can aid inselecting the best method for a specific segmentation goal

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was financially supported by IMDI Grant104002002 (Brainbox) fromZonMw the Netherlands Organ-isation for Health Research and Development within kindsponsoring by Philips theUniversityMedical CenterUtrechtand Eindhoven University of TechnologyThe authors wouldlike to acknowledge the following members of the UtrechtVascular Cognitive Impairment Study Group who were notincluded as coauthors of this paper but were involved in therecruitment of study participants and MRI acquisition at theUMC Utrecht (in alphabetical order by department) E vanden Berg M Brundel S Heringa and L J Kappelle of theDepartment of Neurology P R Luijten and W P Th MMali of the Department of Radiology and A Algra and GE H M Rutten of the Julius Center for Health Sciences andPrimary CareThe research of Geert Jan Biessels and the VCIgroupwas financially supported byVIDIGrant 91711384 fromZonMw and by Grant 2010T073 of the Netherlands HeartFoundation The research of Jeroen de Bresser is financiallysupported by a research talent fellowship of the Univer-sity Medical Center Utrecht (Netherlands) The research ofAnnegreet vanOpbroek andMarleen de Bruijne is financiallysupported by a research grant from NWO (the NetherlandsOrganisation for Scientific Research) The authors would

like to acknowledge MeVis Medical Solutions AG (BremenGermany) for providing MeVisLab Duygu Sarikaya andLiang Zhao acknowledge their Advisor Professor JasonCorsofor his guidance Duygu Sarikaya is supported by NIH 1 R21CA160825-01 and Liang Zhao is partially supported by theChina Scholarship Council (CSC)

References

[1] I Driscoll C Davatzikos Y An et al ldquoLongitudinal pattern ofregional brain volume change differentiates normal aging fromMCIrdquo Neurology vol 72 no 22 pp 1906ndash1913 2009

[2] K Franke and C Gaser ldquoLongitudinal changes in individualBrainAGE in healthy aging mild cognitive impairment andAlzheimerrsquos diseaserdquoGeroPsych vol 25 no 4 pp 235ndash245 2012

[3] MA IkramH A VroomanMWVernooij et al ldquoBrain tissuevolumes in the general elderly populationThe Rotterdam ScanStudyrdquo Neurobiology of Aging vol 29 no 6 pp 882ndash890 2008

[4] A Giorgio and N de Stefano ldquoClinical use of brain volumetryrdquoJournal of Magnetic Resonance Imaging vol 37 no 1 pp 1ndash142013

[5] M W Vannier R L Butterfield D Jordan W A Murphy RG Levitt and M Gado ldquoMultispectral analysis of magneticresonance imagesrdquo Radiology vol 154 no 1 pp 221ndash224 1985

[6] M A Balafar A R Ramli M I Saripan and S MashohorldquoReview of brain MRI image segmentation methodsrdquo ArtificialIntelligence Review vol 33 no 3 pp 261ndash274 2010

[7] J C Bezdek L O Hall and L P Clarke ldquoReview of MR imagesegmentation techniques using pattern recognitionrdquo MedicalPhysics vol 20 no 4 pp 1033ndash1048 1993

[8] L P Clarke R P Velthuizen M A Camacho et al ldquoMRIsegmentation methods and applicationsrdquo Magnetic ResonanceImaging vol 13 no 3 pp 343ndash368 1995

[9] J S Duncan X Papademetris J Yang M Jackowski X Zengand L H Staib ldquoGeometric strategies for neuroanatomic ana-lysis from MRIrdquo NeuroImage vol 23 supplement 1 pp S34ndashS45 2004

[10] D L PhamC Xu and J L Prince ldquoCurrentmethods inmedicalimage segmentationrdquoAnnual Review of Biomedical Engineeringvol 2 no 2000 pp 315ndash337 2000

[11] N Saeed ldquoMagnetic resonance image segmentation usingpattern recognition and applied to image registration andquantitationrdquo NMR in Biomedicine vol 11 no 4-5 pp 157ndash1671998

[12] J S Suri S Singh and L Reden ldquoComputer vision andpattern recognition techniques for 2-D and 3-D MR cerebralcortical segmentation (Part I) a state-of-the-art reviewrdquo PatternAnalysis amp Applications vol 5 no 1 pp 46ndash76 2002

[13] A P Zijdenbos and B M Dawant ldquoBrain segmentation andwhite matter lesion detection in MR imagesrdquo Critical Reviewsin Biomedical Engineering vol 22 no 5-6 pp 401ndash465 1994

[14] K Price ldquoAnything you can do I can do better (No you canrsquot)rdquoComputer Vision Graphics and Image Processing vol 36 no 2-3pp 387ndash391 1986

[15] R de Boer H A Vrooman M A Ikram et al ldquoAccuracy andreproducibility study of automatic MRI brain tissue segmenta-tion methodsrdquo NeuroImage vol 51 no 3 pp 1047ndash1056 2010

[16] J de Bresser M P Portegies A Leemans G J Biessels L JKappelle andM A Viergever ldquoA comparison ofMR based seg-mentation methods for measuring brain atrophy progressionrdquoNeuroImage vol 54 no 2 pp 760ndash768 2011

Computational Intelligence and Neuroscience 15

[17] M B Cuadra L Cammoun T Butz O Cuisenaire and J-P Thiran ldquoComparison and validation of tissue modelizationand statistical classification methods in T1-weighted MR brainimagesrdquo IEEE Transactions on Medical Imaging vol 24 no 12pp 1548ndash1565 2005

[18] F Klauschen A Goldman V Barra A Meyer-Lindenbergand A Lundervold ldquoEvaluation of automated brain MR imagesegmentation and volumetrymethodsrdquoHuman BrainMappingvol 30 no 4 pp 1310ndash1327 2009

[19] H Zaidi T Ruest F Schoenahl and M-L Montandon ldquoCom-parative assessment of statistical brain MR image segmentationalgorithms and their impact on partial volume correction inPETrdquo NeuroImage vol 32 no 4 pp 1591ndash1607 2006

[20] J A Frazier S M Hodge J L Breeze et al ldquoDiagnostic and sexeffects on limbic volumes in early-onset bipolar disorder andschizophreniardquo Schizophrenia Bulletin vol 34 no 1 pp 37ndash462008

[21] A Klein and J Tourville ldquo101 labeled brain images and aconsistent human cortical labeling protocolrdquo Frontiers in Neu-roscience vol 6 article 171 2012

[22] B Van Ginneken T Heimann and M Styner ldquo3D segmen-tation in the clinic a grand challengerdquo in Proceedings of the10th International Conference on Medical Image Computingand Computer Assisted Intervention (MICCAI rsquo07) pp 7ndash15Brisbane Australia October-November 2007

[23] A M Dale B Fischl and M I Sereno ldquoCortical surface-based analysis I Segmentation and surface reconstructionrdquoNeuroImage vol 9 no 2 pp 179ndash194 1999

[24] B Fischl M I Sereno and A M Dale ldquoCortical surface-basedanalysis II inflation flattening and a surface-based coordinatesystemrdquo NeuroImage vol 9 no 2 pp 195ndash207 1999

[25] M Jenkinson C F Beckmann T E J BehrensMWWoolrichand S M Smith ldquoFSLrdquoNeuroImage vol 62 no 2 pp 782ndash7902012

[26] J Ashburner and K J Friston ldquoUnified segmentationrdquo Neu-roImage vol 26 no 3 pp 839ndash851 2005

[27] Y D Reijmer A Leemans M Brundel L J Kappelle and GJ Biessels ldquoDisruption of the cerebral white matter network isrelated to slowing of information processing speed in patientswith type 2 diabetesrdquoDiabetes vol 62 no 6 pp 2112ndash2115 2013

[28] S Klein M Staring K Murphy M A Viergever and J P WPluim ldquoElastix a toolbox for intensity-based medical imageregistrationrdquo IEEE Transactions onMedical Imaging vol 29 no1 pp 196ndash205 2010

[29] K J Friston J T Ashburner S J Kiebel T E Nichols and WD Penny Eds Statistical Parametric Mapping The Analysis ofFunctional Brain Images Academic Press 2011

[30] F Ritter T Boskamp A Homeyer et al ldquoMedical imageanalysis a visual approachrdquo IEEE Pulse vol 2 no 6 pp 60ndash702011

[31] L R Dice ldquoMeasures of the amount of ecologic associationbetween speciesrdquo Ecology vol 26 no 3 pp 297ndash302 1945

[32] D P Huttenlocher G A Klanderman and W J RucklidgeldquoComparing images using the Hausdorff distancerdquo IEEE Trans-actions on Pattern Analysis and Machine Intelligence vol 15 no9 pp 850ndash863 1993

[33] A Alansary A Soliman F Khalifa et al ldquoMAP-based frame-work for segmentation of MR brain images based on visualappearance and prior shaperdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal September 2013

[34] A Jog S Roy J L Prince and A Carass ldquoMR brain segmenta-tion using decision treesrdquo in Proceedings of the MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal Nagoya Japan September 2013

[35] R Katyal S Paneri and M Kuse ldquoGaussian intensity modelwith neighborhood cues for fluid-tissue categorization ofmulti-sequence MR brain imagesrdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal Nagoya Japan September 2013

[36] QMahmoodMAlipoor A Chodorowski AMehnert andMPersson ldquoMultimodal MR brain segmentation using Bayesian-based adaptive mean-shift (BAMS)rdquo in Proceedings of theMICCAI WorkshopsmdashThe MICCAI Grand Challenge on MRBrain Image Segmentation (MRBrainS rsquo13) The Midas JournalSeptember 2013

[37] A van Opbroek F van der Lijn andM de Bruijne ldquoAutomatedbrain-tissue segmentation by multi-feature SVM classificationrdquoin Proceedings of the MICCAI WorkshopsmdashThe MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal September 2013

[38] S Pereira J Festa J A Mariz N Sousa and C A SilvaldquoAutomatic brain tissue segmentation of multi-sequence MRimages using random decision forestsrdquo in Proceedings of theMICCAI Grand Challenge on MR Brain Image Segmentation(MRBrainS rsquo13) The Midas Journal Nagoya Japan September2013

[39] MRajchl J Baxter J Yuan T Peters andAKhan ldquoMulti-atlas-based segmentation with hierarchical max-flowrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[40] D Sarikaya L Zhao and J J Corso ldquoMulti-atlas brain MRIsegmentation with multiway cutrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[41] H Vrooman F van der Lijn and W Niessen ldquoAuto-kNNbrain tissue segmentation using automatically trained k-nearest-neighbor classificationrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[42] S Vyas P Burlina D Kleissas and R Mukherjee ldquoAutomatedwalks using machine learning for segmentationrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[43] C Wang and O Smedby ldquoFully automatic brain segmentationusing model-guided level sets and skeleton-based modelsrdquo inProceedings of theMICCAI Grand Challenge onMRBrain ImageSegmentation (MRBrainS rsquo13) The Midas Journal NagoyaJapan September 2013

[44] C Cortes and V Vapnik ldquoSupport-vector networksrdquo MachineLearning vol 20 no 3 pp 273ndash297 1995

[45] H A Vrooman C A Cocosco F van der Lijn et al ldquoMulti-spectral brain tissue segmentation using automatically trainedk-Nearest-Neighbor classificationrdquo NeuroImage vol 37 no 1pp 71ndash81 2007

[46] C Wang H Frimmel and O Smedby ldquoFast level-set basedimage segmentation using coherent propagationrdquo MedicalPhysics vol 41 no 7 Article ID 073501 2014

16 Computational Intelligence and Neuroscience

[47] A Jog S Roy A Carass and J L Prince ldquoMagnetic resonanceimage synthesis through patch regressionrdquo in Proceedings of theIEEE 10th International Symposium onBiomedical Imaging (ISBIrsquo13) pp 350ndash353 April 2013

[48] J G Sied A P Zijdenbos and A C Evans ldquoA non-parametricmethod for automatic correction of intensity non-uniformity inMRI datardquo IEEE Transactions on Medical Imaging vol 17 no 1pp 87ndash97 1998

[49] A Carass J Cuzzocreo M B Wheeler P-L Bazin S MResnick and J L Prince ldquoSimple paradigm for extra-cerebraltissue removal algorithm and analysisrdquoNeuroImage vol 56 no4 pp 1982ndash1992 2011

[50] M Modat G R Ridgway Z A Taylor et al ldquoFast free-form deformation using graphics processing unitsrdquo ComputerMethods and Programs in Biomedicine vol 98 no 3 pp 278ndash284 2010

[51] M Rajchl J Yuan J A White et al ldquoInteractive hierarchical-flow segmentation of scar tissue from late-enhancement cardiacmr imagesrdquo IEEE Transactions on Medical Imaging vol 33 no1 pp 159ndash172 2014

[52] QMahmood A Chodorowski AMehnert andM Persson ldquoAnovel Bayesian approach to adaptive mean shift segmentationof brain imagesrdquo in Proceedings of the 25th IEEE InternationalSymposiumonComputer-BasedMedical Systems (CBMS rsquo12) pp1ndash6 June 2012

[53] A El-Baz M Nitzken A Alansary A Soliman and MCasanova ldquoBrain Segmentation Method for Young Childrenand Adultsrdquo US Provisional Patent Application 13100

[54] A Alansary A Soliman M Nitzken et al ldquoAn integratedgeometrical and stochastic approach for accurate infant brainextractionrdquo in Proceedings of the IEEE International Conferenceon Image Processing (ICIP rsquo14) pp 3542ndash3546 October 2014

[55] V Popescu M Battaglini W S Hoogstrate et al ldquoOptimizingparameter choice for FSL-Brain Extraction Tool (BET) on 3DT1 images in multiple sclerosisrdquo NeuroImage vol 61 no 4 pp1484ndash1494 2012

[56] J L Whitwell W R Crum H C Watt and N C FoxldquoNormalization of cerebral volumes by use of intracranial vol-ume implications for longitudinal quantitative MR imagingrdquoAmerican Journal of Neuroradiology vol 22 no 8 pp 1483ndash1489 2001

[57] S M Smith ldquoFast robust automated brain extractionrdquo HumanBrain Mapping vol 17 no 3 pp 143ndash155 2002

[58] D W Shattuck G Prasad M Mirza K L Narr and A WToga ldquoOnline resource for validation of brain segmentationmethodsrdquo NeuroImage vol 45 no 2 pp 431ndash439 2009

[59] I Isgum M J N L Benders B Avants et al ldquoEvaluation ofautomatic neonatal brain segmentation algorithms the Neo-BrainS12 challengerdquo Medical Image Analysis vol 20 no 1 pp135ndash151 2015

[60] P Moeskops M J N L Benders S M Chita et al ldquoAutomaticsegmentation of MR brain images of preterm infants usingsupervised classificationrdquo NeuroImage 2015

[61] P Moeskops M A Viergever M J N L Benders and IIsgum ldquoEvaluation of an automatic brain segmentationmethoddeveloped for neonates on adult MR brain imagesrdquo in MedicalImaging Image Processing vol 9413 of Proceedings of SPIEInternational Society for Optics and Photonics Orlando FlaUSA February 2015

[62] L Wang Y Gao F Shi et al ldquoLINKS learning-based multi-source IntegratioN frameworK for Segmentation of infant brainimagesrdquo NeuroImage vol 108 pp 160ndash172 2015

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience 13

T2

UofL BioIm

Nars

FSL‐S

S2_Q

‐FLAIR

aging (860)

il (892)

eg (453)

M (565)

T1‐IR

CMIV (90

SPM12_T1_F

SPM12_T1_IR

LNMBrains

5) UB

(831) SP

_F (815) F

(286)

T1

VPML Med (9

M12_T1_IR (60

SL‐PVSeg (525

R

10) Bigr_

5) MN

) FreeS

eference

neuro (896)

AB (889)

urfer (846)

BIGR2

Robarts

SPM12_T

Jedi Mind M

(852)

(810)

1 (683)

eld (769)

Figure 6 Illustration of the segmentation results in the presence of whitematter lesions (test subject 3 slice 31)Whitematter lesions (WMLs)should be labeled asWM the sensitivity (percentage ofWML voxels is labeled asWM) over all 15 datasets is presented between brackets afterthe team nameThe first three images show the threeMRI sequences T2-weighted fluid attenuated inversion recovery T1-weighted inversionrecovery and T1-weighted scanThe fourth image (reference) is themanually segmented reference standard (red whitematter lesions yellowwhite matter light blue gray matter and dark blue cerebrospinal fluid)The results of the segmentation algorithms are ordered from the bestoverall performance (BIGR2) to the worst overall performance (LNMBrains)The arrows indicate example locations where the segmentationresults differ from the ground truth

14 Computational Intelligence and Neuroscience

fromboth theMRBrainS and theNeoBrainS [59] (brain tissuesegmentation in neonates) challenge Two methods [60ndash62]specifically designed for neonatal brain tissue segmentationshowed a high performance for tissue segmentation on theMRBrainS data Applying algorithms to different types of datahas the potential to lead to new insights and more robustalgorithms The MRBrainS evaluation framework remainsopen for new contributions At the time of writing 21teams had submitted their results on the MRBrainS website(httpmrbrains13isiuunlresultsphp)

5 Conclusion

The MRBrainS challenge online evaluation framework pro-vides direct and objective comparison of automatic andsemiautomatic methods to segment GM WM CSF brainand ICV on 3T multisequence MRI data The first elevenparticipating methods are evaluated and presented in thispaper as well as three commonly used freeware packages(FreeSurfer FSL and SPM12) They provide a benchmarkfor future contributions to the framework The final rankingprovides a quick insight into the overall performance of theevaluated methods in comparison to each other whereasthe individual evaluation measures (Dice 95th-percentileHausdorff distance and absolute volume difference) percomponent (GM WM CSF brain and ICV) can aid inselecting the best method for a specific segmentation goal

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was financially supported by IMDI Grant104002002 (Brainbox) fromZonMw the Netherlands Organ-isation for Health Research and Development within kindsponsoring by Philips theUniversityMedical CenterUtrechtand Eindhoven University of TechnologyThe authors wouldlike to acknowledge the following members of the UtrechtVascular Cognitive Impairment Study Group who were notincluded as coauthors of this paper but were involved in therecruitment of study participants and MRI acquisition at theUMC Utrecht (in alphabetical order by department) E vanden Berg M Brundel S Heringa and L J Kappelle of theDepartment of Neurology P R Luijten and W P Th MMali of the Department of Radiology and A Algra and GE H M Rutten of the Julius Center for Health Sciences andPrimary CareThe research of Geert Jan Biessels and the VCIgroupwas financially supported byVIDIGrant 91711384 fromZonMw and by Grant 2010T073 of the Netherlands HeartFoundation The research of Jeroen de Bresser is financiallysupported by a research talent fellowship of the Univer-sity Medical Center Utrecht (Netherlands) The research ofAnnegreet vanOpbroek andMarleen de Bruijne is financiallysupported by a research grant from NWO (the NetherlandsOrganisation for Scientific Research) The authors would

like to acknowledge MeVis Medical Solutions AG (BremenGermany) for providing MeVisLab Duygu Sarikaya andLiang Zhao acknowledge their Advisor Professor JasonCorsofor his guidance Duygu Sarikaya is supported by NIH 1 R21CA160825-01 and Liang Zhao is partially supported by theChina Scholarship Council (CSC)

References

[1] I Driscoll C Davatzikos Y An et al ldquoLongitudinal pattern ofregional brain volume change differentiates normal aging fromMCIrdquo Neurology vol 72 no 22 pp 1906ndash1913 2009

[2] K Franke and C Gaser ldquoLongitudinal changes in individualBrainAGE in healthy aging mild cognitive impairment andAlzheimerrsquos diseaserdquoGeroPsych vol 25 no 4 pp 235ndash245 2012

[3] MA IkramH A VroomanMWVernooij et al ldquoBrain tissuevolumes in the general elderly populationThe Rotterdam ScanStudyrdquo Neurobiology of Aging vol 29 no 6 pp 882ndash890 2008

[4] A Giorgio and N de Stefano ldquoClinical use of brain volumetryrdquoJournal of Magnetic Resonance Imaging vol 37 no 1 pp 1ndash142013

[5] M W Vannier R L Butterfield D Jordan W A Murphy RG Levitt and M Gado ldquoMultispectral analysis of magneticresonance imagesrdquo Radiology vol 154 no 1 pp 221ndash224 1985

[6] M A Balafar A R Ramli M I Saripan and S MashohorldquoReview of brain MRI image segmentation methodsrdquo ArtificialIntelligence Review vol 33 no 3 pp 261ndash274 2010

[7] J C Bezdek L O Hall and L P Clarke ldquoReview of MR imagesegmentation techniques using pattern recognitionrdquo MedicalPhysics vol 20 no 4 pp 1033ndash1048 1993

[8] L P Clarke R P Velthuizen M A Camacho et al ldquoMRIsegmentation methods and applicationsrdquo Magnetic ResonanceImaging vol 13 no 3 pp 343ndash368 1995

[9] J S Duncan X Papademetris J Yang M Jackowski X Zengand L H Staib ldquoGeometric strategies for neuroanatomic ana-lysis from MRIrdquo NeuroImage vol 23 supplement 1 pp S34ndashS45 2004

[10] D L PhamC Xu and J L Prince ldquoCurrentmethods inmedicalimage segmentationrdquoAnnual Review of Biomedical Engineeringvol 2 no 2000 pp 315ndash337 2000

[11] N Saeed ldquoMagnetic resonance image segmentation usingpattern recognition and applied to image registration andquantitationrdquo NMR in Biomedicine vol 11 no 4-5 pp 157ndash1671998

[12] J S Suri S Singh and L Reden ldquoComputer vision andpattern recognition techniques for 2-D and 3-D MR cerebralcortical segmentation (Part I) a state-of-the-art reviewrdquo PatternAnalysis amp Applications vol 5 no 1 pp 46ndash76 2002

[13] A P Zijdenbos and B M Dawant ldquoBrain segmentation andwhite matter lesion detection in MR imagesrdquo Critical Reviewsin Biomedical Engineering vol 22 no 5-6 pp 401ndash465 1994

[14] K Price ldquoAnything you can do I can do better (No you canrsquot)rdquoComputer Vision Graphics and Image Processing vol 36 no 2-3pp 387ndash391 1986

[15] R de Boer H A Vrooman M A Ikram et al ldquoAccuracy andreproducibility study of automatic MRI brain tissue segmenta-tion methodsrdquo NeuroImage vol 51 no 3 pp 1047ndash1056 2010

[16] J de Bresser M P Portegies A Leemans G J Biessels L JKappelle andM A Viergever ldquoA comparison ofMR based seg-mentation methods for measuring brain atrophy progressionrdquoNeuroImage vol 54 no 2 pp 760ndash768 2011

Computational Intelligence and Neuroscience 15

[17] M B Cuadra L Cammoun T Butz O Cuisenaire and J-P Thiran ldquoComparison and validation of tissue modelizationand statistical classification methods in T1-weighted MR brainimagesrdquo IEEE Transactions on Medical Imaging vol 24 no 12pp 1548ndash1565 2005

[18] F Klauschen A Goldman V Barra A Meyer-Lindenbergand A Lundervold ldquoEvaluation of automated brain MR imagesegmentation and volumetrymethodsrdquoHuman BrainMappingvol 30 no 4 pp 1310ndash1327 2009

[19] H Zaidi T Ruest F Schoenahl and M-L Montandon ldquoCom-parative assessment of statistical brain MR image segmentationalgorithms and their impact on partial volume correction inPETrdquo NeuroImage vol 32 no 4 pp 1591ndash1607 2006

[20] J A Frazier S M Hodge J L Breeze et al ldquoDiagnostic and sexeffects on limbic volumes in early-onset bipolar disorder andschizophreniardquo Schizophrenia Bulletin vol 34 no 1 pp 37ndash462008

[21] A Klein and J Tourville ldquo101 labeled brain images and aconsistent human cortical labeling protocolrdquo Frontiers in Neu-roscience vol 6 article 171 2012

[22] B Van Ginneken T Heimann and M Styner ldquo3D segmen-tation in the clinic a grand challengerdquo in Proceedings of the10th International Conference on Medical Image Computingand Computer Assisted Intervention (MICCAI rsquo07) pp 7ndash15Brisbane Australia October-November 2007

[23] A M Dale B Fischl and M I Sereno ldquoCortical surface-based analysis I Segmentation and surface reconstructionrdquoNeuroImage vol 9 no 2 pp 179ndash194 1999

[24] B Fischl M I Sereno and A M Dale ldquoCortical surface-basedanalysis II inflation flattening and a surface-based coordinatesystemrdquo NeuroImage vol 9 no 2 pp 195ndash207 1999

[25] M Jenkinson C F Beckmann T E J BehrensMWWoolrichand S M Smith ldquoFSLrdquoNeuroImage vol 62 no 2 pp 782ndash7902012

[26] J Ashburner and K J Friston ldquoUnified segmentationrdquo Neu-roImage vol 26 no 3 pp 839ndash851 2005

[27] Y D Reijmer A Leemans M Brundel L J Kappelle and GJ Biessels ldquoDisruption of the cerebral white matter network isrelated to slowing of information processing speed in patientswith type 2 diabetesrdquoDiabetes vol 62 no 6 pp 2112ndash2115 2013

[28] S Klein M Staring K Murphy M A Viergever and J P WPluim ldquoElastix a toolbox for intensity-based medical imageregistrationrdquo IEEE Transactions onMedical Imaging vol 29 no1 pp 196ndash205 2010

[29] K J Friston J T Ashburner S J Kiebel T E Nichols and WD Penny Eds Statistical Parametric Mapping The Analysis ofFunctional Brain Images Academic Press 2011

[30] F Ritter T Boskamp A Homeyer et al ldquoMedical imageanalysis a visual approachrdquo IEEE Pulse vol 2 no 6 pp 60ndash702011

[31] L R Dice ldquoMeasures of the amount of ecologic associationbetween speciesrdquo Ecology vol 26 no 3 pp 297ndash302 1945

[32] D P Huttenlocher G A Klanderman and W J RucklidgeldquoComparing images using the Hausdorff distancerdquo IEEE Trans-actions on Pattern Analysis and Machine Intelligence vol 15 no9 pp 850ndash863 1993

[33] A Alansary A Soliman F Khalifa et al ldquoMAP-based frame-work for segmentation of MR brain images based on visualappearance and prior shaperdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal September 2013

[34] A Jog S Roy J L Prince and A Carass ldquoMR brain segmenta-tion using decision treesrdquo in Proceedings of the MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal Nagoya Japan September 2013

[35] R Katyal S Paneri and M Kuse ldquoGaussian intensity modelwith neighborhood cues for fluid-tissue categorization ofmulti-sequence MR brain imagesrdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal Nagoya Japan September 2013

[36] QMahmoodMAlipoor A Chodorowski AMehnert andMPersson ldquoMultimodal MR brain segmentation using Bayesian-based adaptive mean-shift (BAMS)rdquo in Proceedings of theMICCAI WorkshopsmdashThe MICCAI Grand Challenge on MRBrain Image Segmentation (MRBrainS rsquo13) The Midas JournalSeptember 2013

[37] A van Opbroek F van der Lijn andM de Bruijne ldquoAutomatedbrain-tissue segmentation by multi-feature SVM classificationrdquoin Proceedings of the MICCAI WorkshopsmdashThe MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal September 2013

[38] S Pereira J Festa J A Mariz N Sousa and C A SilvaldquoAutomatic brain tissue segmentation of multi-sequence MRimages using random decision forestsrdquo in Proceedings of theMICCAI Grand Challenge on MR Brain Image Segmentation(MRBrainS rsquo13) The Midas Journal Nagoya Japan September2013

[39] MRajchl J Baxter J Yuan T Peters andAKhan ldquoMulti-atlas-based segmentation with hierarchical max-flowrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[40] D Sarikaya L Zhao and J J Corso ldquoMulti-atlas brain MRIsegmentation with multiway cutrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[41] H Vrooman F van der Lijn and W Niessen ldquoAuto-kNNbrain tissue segmentation using automatically trained k-nearest-neighbor classificationrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[42] S Vyas P Burlina D Kleissas and R Mukherjee ldquoAutomatedwalks using machine learning for segmentationrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[43] C Wang and O Smedby ldquoFully automatic brain segmentationusing model-guided level sets and skeleton-based modelsrdquo inProceedings of theMICCAI Grand Challenge onMRBrain ImageSegmentation (MRBrainS rsquo13) The Midas Journal NagoyaJapan September 2013

[44] C Cortes and V Vapnik ldquoSupport-vector networksrdquo MachineLearning vol 20 no 3 pp 273ndash297 1995

[45] H A Vrooman C A Cocosco F van der Lijn et al ldquoMulti-spectral brain tissue segmentation using automatically trainedk-Nearest-Neighbor classificationrdquo NeuroImage vol 37 no 1pp 71ndash81 2007

[46] C Wang H Frimmel and O Smedby ldquoFast level-set basedimage segmentation using coherent propagationrdquo MedicalPhysics vol 41 no 7 Article ID 073501 2014

16 Computational Intelligence and Neuroscience

[47] A Jog S Roy A Carass and J L Prince ldquoMagnetic resonanceimage synthesis through patch regressionrdquo in Proceedings of theIEEE 10th International Symposium onBiomedical Imaging (ISBIrsquo13) pp 350ndash353 April 2013

[48] J G Sied A P Zijdenbos and A C Evans ldquoA non-parametricmethod for automatic correction of intensity non-uniformity inMRI datardquo IEEE Transactions on Medical Imaging vol 17 no 1pp 87ndash97 1998

[49] A Carass J Cuzzocreo M B Wheeler P-L Bazin S MResnick and J L Prince ldquoSimple paradigm for extra-cerebraltissue removal algorithm and analysisrdquoNeuroImage vol 56 no4 pp 1982ndash1992 2011

[50] M Modat G R Ridgway Z A Taylor et al ldquoFast free-form deformation using graphics processing unitsrdquo ComputerMethods and Programs in Biomedicine vol 98 no 3 pp 278ndash284 2010

[51] M Rajchl J Yuan J A White et al ldquoInteractive hierarchical-flow segmentation of scar tissue from late-enhancement cardiacmr imagesrdquo IEEE Transactions on Medical Imaging vol 33 no1 pp 159ndash172 2014

[52] QMahmood A Chodorowski AMehnert andM Persson ldquoAnovel Bayesian approach to adaptive mean shift segmentationof brain imagesrdquo in Proceedings of the 25th IEEE InternationalSymposiumonComputer-BasedMedical Systems (CBMS rsquo12) pp1ndash6 June 2012

[53] A El-Baz M Nitzken A Alansary A Soliman and MCasanova ldquoBrain Segmentation Method for Young Childrenand Adultsrdquo US Provisional Patent Application 13100

[54] A Alansary A Soliman M Nitzken et al ldquoAn integratedgeometrical and stochastic approach for accurate infant brainextractionrdquo in Proceedings of the IEEE International Conferenceon Image Processing (ICIP rsquo14) pp 3542ndash3546 October 2014

[55] V Popescu M Battaglini W S Hoogstrate et al ldquoOptimizingparameter choice for FSL-Brain Extraction Tool (BET) on 3DT1 images in multiple sclerosisrdquo NeuroImage vol 61 no 4 pp1484ndash1494 2012

[56] J L Whitwell W R Crum H C Watt and N C FoxldquoNormalization of cerebral volumes by use of intracranial vol-ume implications for longitudinal quantitative MR imagingrdquoAmerican Journal of Neuroradiology vol 22 no 8 pp 1483ndash1489 2001

[57] S M Smith ldquoFast robust automated brain extractionrdquo HumanBrain Mapping vol 17 no 3 pp 143ndash155 2002

[58] D W Shattuck G Prasad M Mirza K L Narr and A WToga ldquoOnline resource for validation of brain segmentationmethodsrdquo NeuroImage vol 45 no 2 pp 431ndash439 2009

[59] I Isgum M J N L Benders B Avants et al ldquoEvaluation ofautomatic neonatal brain segmentation algorithms the Neo-BrainS12 challengerdquo Medical Image Analysis vol 20 no 1 pp135ndash151 2015

[60] P Moeskops M J N L Benders S M Chita et al ldquoAutomaticsegmentation of MR brain images of preterm infants usingsupervised classificationrdquo NeuroImage 2015

[61] P Moeskops M A Viergever M J N L Benders and IIsgum ldquoEvaluation of an automatic brain segmentationmethoddeveloped for neonates on adult MR brain imagesrdquo in MedicalImaging Image Processing vol 9413 of Proceedings of SPIEInternational Society for Optics and Photonics Orlando FlaUSA February 2015

[62] L Wang Y Gao F Shi et al ldquoLINKS learning-based multi-source IntegratioN frameworK for Segmentation of infant brainimagesrdquo NeuroImage vol 108 pp 160ndash172 2015

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

14 Computational Intelligence and Neuroscience

fromboth theMRBrainS and theNeoBrainS [59] (brain tissuesegmentation in neonates) challenge Two methods [60ndash62]specifically designed for neonatal brain tissue segmentationshowed a high performance for tissue segmentation on theMRBrainS data Applying algorithms to different types of datahas the potential to lead to new insights and more robustalgorithms The MRBrainS evaluation framework remainsopen for new contributions At the time of writing 21teams had submitted their results on the MRBrainS website(httpmrbrains13isiuunlresultsphp)

5 Conclusion

The MRBrainS challenge online evaluation framework pro-vides direct and objective comparison of automatic andsemiautomatic methods to segment GM WM CSF brainand ICV on 3T multisequence MRI data The first elevenparticipating methods are evaluated and presented in thispaper as well as three commonly used freeware packages(FreeSurfer FSL and SPM12) They provide a benchmarkfor future contributions to the framework The final rankingprovides a quick insight into the overall performance of theevaluated methods in comparison to each other whereasthe individual evaluation measures (Dice 95th-percentileHausdorff distance and absolute volume difference) percomponent (GM WM CSF brain and ICV) can aid inselecting the best method for a specific segmentation goal

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was financially supported by IMDI Grant104002002 (Brainbox) fromZonMw the Netherlands Organ-isation for Health Research and Development within kindsponsoring by Philips theUniversityMedical CenterUtrechtand Eindhoven University of TechnologyThe authors wouldlike to acknowledge the following members of the UtrechtVascular Cognitive Impairment Study Group who were notincluded as coauthors of this paper but were involved in therecruitment of study participants and MRI acquisition at theUMC Utrecht (in alphabetical order by department) E vanden Berg M Brundel S Heringa and L J Kappelle of theDepartment of Neurology P R Luijten and W P Th MMali of the Department of Radiology and A Algra and GE H M Rutten of the Julius Center for Health Sciences andPrimary CareThe research of Geert Jan Biessels and the VCIgroupwas financially supported byVIDIGrant 91711384 fromZonMw and by Grant 2010T073 of the Netherlands HeartFoundation The research of Jeroen de Bresser is financiallysupported by a research talent fellowship of the Univer-sity Medical Center Utrecht (Netherlands) The research ofAnnegreet vanOpbroek andMarleen de Bruijne is financiallysupported by a research grant from NWO (the NetherlandsOrganisation for Scientific Research) The authors would

like to acknowledge MeVis Medical Solutions AG (BremenGermany) for providing MeVisLab Duygu Sarikaya andLiang Zhao acknowledge their Advisor Professor JasonCorsofor his guidance Duygu Sarikaya is supported by NIH 1 R21CA160825-01 and Liang Zhao is partially supported by theChina Scholarship Council (CSC)

References

[1] I Driscoll C Davatzikos Y An et al ldquoLongitudinal pattern ofregional brain volume change differentiates normal aging fromMCIrdquo Neurology vol 72 no 22 pp 1906ndash1913 2009

[2] K Franke and C Gaser ldquoLongitudinal changes in individualBrainAGE in healthy aging mild cognitive impairment andAlzheimerrsquos diseaserdquoGeroPsych vol 25 no 4 pp 235ndash245 2012

[3] MA IkramH A VroomanMWVernooij et al ldquoBrain tissuevolumes in the general elderly populationThe Rotterdam ScanStudyrdquo Neurobiology of Aging vol 29 no 6 pp 882ndash890 2008

[4] A Giorgio and N de Stefano ldquoClinical use of brain volumetryrdquoJournal of Magnetic Resonance Imaging vol 37 no 1 pp 1ndash142013

[5] M W Vannier R L Butterfield D Jordan W A Murphy RG Levitt and M Gado ldquoMultispectral analysis of magneticresonance imagesrdquo Radiology vol 154 no 1 pp 221ndash224 1985

[6] M A Balafar A R Ramli M I Saripan and S MashohorldquoReview of brain MRI image segmentation methodsrdquo ArtificialIntelligence Review vol 33 no 3 pp 261ndash274 2010

[7] J C Bezdek L O Hall and L P Clarke ldquoReview of MR imagesegmentation techniques using pattern recognitionrdquo MedicalPhysics vol 20 no 4 pp 1033ndash1048 1993

[8] L P Clarke R P Velthuizen M A Camacho et al ldquoMRIsegmentation methods and applicationsrdquo Magnetic ResonanceImaging vol 13 no 3 pp 343ndash368 1995

[9] J S Duncan X Papademetris J Yang M Jackowski X Zengand L H Staib ldquoGeometric strategies for neuroanatomic ana-lysis from MRIrdquo NeuroImage vol 23 supplement 1 pp S34ndashS45 2004

[10] D L PhamC Xu and J L Prince ldquoCurrentmethods inmedicalimage segmentationrdquoAnnual Review of Biomedical Engineeringvol 2 no 2000 pp 315ndash337 2000

[11] N Saeed ldquoMagnetic resonance image segmentation usingpattern recognition and applied to image registration andquantitationrdquo NMR in Biomedicine vol 11 no 4-5 pp 157ndash1671998

[12] J S Suri S Singh and L Reden ldquoComputer vision andpattern recognition techniques for 2-D and 3-D MR cerebralcortical segmentation (Part I) a state-of-the-art reviewrdquo PatternAnalysis amp Applications vol 5 no 1 pp 46ndash76 2002

[13] A P Zijdenbos and B M Dawant ldquoBrain segmentation andwhite matter lesion detection in MR imagesrdquo Critical Reviewsin Biomedical Engineering vol 22 no 5-6 pp 401ndash465 1994

[14] K Price ldquoAnything you can do I can do better (No you canrsquot)rdquoComputer Vision Graphics and Image Processing vol 36 no 2-3pp 387ndash391 1986

[15] R de Boer H A Vrooman M A Ikram et al ldquoAccuracy andreproducibility study of automatic MRI brain tissue segmenta-tion methodsrdquo NeuroImage vol 51 no 3 pp 1047ndash1056 2010

[16] J de Bresser M P Portegies A Leemans G J Biessels L JKappelle andM A Viergever ldquoA comparison ofMR based seg-mentation methods for measuring brain atrophy progressionrdquoNeuroImage vol 54 no 2 pp 760ndash768 2011

Computational Intelligence and Neuroscience 15

[17] M B Cuadra L Cammoun T Butz O Cuisenaire and J-P Thiran ldquoComparison and validation of tissue modelizationand statistical classification methods in T1-weighted MR brainimagesrdquo IEEE Transactions on Medical Imaging vol 24 no 12pp 1548ndash1565 2005

[18] F Klauschen A Goldman V Barra A Meyer-Lindenbergand A Lundervold ldquoEvaluation of automated brain MR imagesegmentation and volumetrymethodsrdquoHuman BrainMappingvol 30 no 4 pp 1310ndash1327 2009

[19] H Zaidi T Ruest F Schoenahl and M-L Montandon ldquoCom-parative assessment of statistical brain MR image segmentationalgorithms and their impact on partial volume correction inPETrdquo NeuroImage vol 32 no 4 pp 1591ndash1607 2006

[20] J A Frazier S M Hodge J L Breeze et al ldquoDiagnostic and sexeffects on limbic volumes in early-onset bipolar disorder andschizophreniardquo Schizophrenia Bulletin vol 34 no 1 pp 37ndash462008

[21] A Klein and J Tourville ldquo101 labeled brain images and aconsistent human cortical labeling protocolrdquo Frontiers in Neu-roscience vol 6 article 171 2012

[22] B Van Ginneken T Heimann and M Styner ldquo3D segmen-tation in the clinic a grand challengerdquo in Proceedings of the10th International Conference on Medical Image Computingand Computer Assisted Intervention (MICCAI rsquo07) pp 7ndash15Brisbane Australia October-November 2007

[23] A M Dale B Fischl and M I Sereno ldquoCortical surface-based analysis I Segmentation and surface reconstructionrdquoNeuroImage vol 9 no 2 pp 179ndash194 1999

[24] B Fischl M I Sereno and A M Dale ldquoCortical surface-basedanalysis II inflation flattening and a surface-based coordinatesystemrdquo NeuroImage vol 9 no 2 pp 195ndash207 1999

[25] M Jenkinson C F Beckmann T E J BehrensMWWoolrichand S M Smith ldquoFSLrdquoNeuroImage vol 62 no 2 pp 782ndash7902012

[26] J Ashburner and K J Friston ldquoUnified segmentationrdquo Neu-roImage vol 26 no 3 pp 839ndash851 2005

[27] Y D Reijmer A Leemans M Brundel L J Kappelle and GJ Biessels ldquoDisruption of the cerebral white matter network isrelated to slowing of information processing speed in patientswith type 2 diabetesrdquoDiabetes vol 62 no 6 pp 2112ndash2115 2013

[28] S Klein M Staring K Murphy M A Viergever and J P WPluim ldquoElastix a toolbox for intensity-based medical imageregistrationrdquo IEEE Transactions onMedical Imaging vol 29 no1 pp 196ndash205 2010

[29] K J Friston J T Ashburner S J Kiebel T E Nichols and WD Penny Eds Statistical Parametric Mapping The Analysis ofFunctional Brain Images Academic Press 2011

[30] F Ritter T Boskamp A Homeyer et al ldquoMedical imageanalysis a visual approachrdquo IEEE Pulse vol 2 no 6 pp 60ndash702011

[31] L R Dice ldquoMeasures of the amount of ecologic associationbetween speciesrdquo Ecology vol 26 no 3 pp 297ndash302 1945

[32] D P Huttenlocher G A Klanderman and W J RucklidgeldquoComparing images using the Hausdorff distancerdquo IEEE Trans-actions on Pattern Analysis and Machine Intelligence vol 15 no9 pp 850ndash863 1993

[33] A Alansary A Soliman F Khalifa et al ldquoMAP-based frame-work for segmentation of MR brain images based on visualappearance and prior shaperdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal September 2013

[34] A Jog S Roy J L Prince and A Carass ldquoMR brain segmenta-tion using decision treesrdquo in Proceedings of the MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal Nagoya Japan September 2013

[35] R Katyal S Paneri and M Kuse ldquoGaussian intensity modelwith neighborhood cues for fluid-tissue categorization ofmulti-sequence MR brain imagesrdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal Nagoya Japan September 2013

[36] QMahmoodMAlipoor A Chodorowski AMehnert andMPersson ldquoMultimodal MR brain segmentation using Bayesian-based adaptive mean-shift (BAMS)rdquo in Proceedings of theMICCAI WorkshopsmdashThe MICCAI Grand Challenge on MRBrain Image Segmentation (MRBrainS rsquo13) The Midas JournalSeptember 2013

[37] A van Opbroek F van der Lijn andM de Bruijne ldquoAutomatedbrain-tissue segmentation by multi-feature SVM classificationrdquoin Proceedings of the MICCAI WorkshopsmdashThe MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal September 2013

[38] S Pereira J Festa J A Mariz N Sousa and C A SilvaldquoAutomatic brain tissue segmentation of multi-sequence MRimages using random decision forestsrdquo in Proceedings of theMICCAI Grand Challenge on MR Brain Image Segmentation(MRBrainS rsquo13) The Midas Journal Nagoya Japan September2013

[39] MRajchl J Baxter J Yuan T Peters andAKhan ldquoMulti-atlas-based segmentation with hierarchical max-flowrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[40] D Sarikaya L Zhao and J J Corso ldquoMulti-atlas brain MRIsegmentation with multiway cutrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[41] H Vrooman F van der Lijn and W Niessen ldquoAuto-kNNbrain tissue segmentation using automatically trained k-nearest-neighbor classificationrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[42] S Vyas P Burlina D Kleissas and R Mukherjee ldquoAutomatedwalks using machine learning for segmentationrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[43] C Wang and O Smedby ldquoFully automatic brain segmentationusing model-guided level sets and skeleton-based modelsrdquo inProceedings of theMICCAI Grand Challenge onMRBrain ImageSegmentation (MRBrainS rsquo13) The Midas Journal NagoyaJapan September 2013

[44] C Cortes and V Vapnik ldquoSupport-vector networksrdquo MachineLearning vol 20 no 3 pp 273ndash297 1995

[45] H A Vrooman C A Cocosco F van der Lijn et al ldquoMulti-spectral brain tissue segmentation using automatically trainedk-Nearest-Neighbor classificationrdquo NeuroImage vol 37 no 1pp 71ndash81 2007

[46] C Wang H Frimmel and O Smedby ldquoFast level-set basedimage segmentation using coherent propagationrdquo MedicalPhysics vol 41 no 7 Article ID 073501 2014

16 Computational Intelligence and Neuroscience

[47] A Jog S Roy A Carass and J L Prince ldquoMagnetic resonanceimage synthesis through patch regressionrdquo in Proceedings of theIEEE 10th International Symposium onBiomedical Imaging (ISBIrsquo13) pp 350ndash353 April 2013

[48] J G Sied A P Zijdenbos and A C Evans ldquoA non-parametricmethod for automatic correction of intensity non-uniformity inMRI datardquo IEEE Transactions on Medical Imaging vol 17 no 1pp 87ndash97 1998

[49] A Carass J Cuzzocreo M B Wheeler P-L Bazin S MResnick and J L Prince ldquoSimple paradigm for extra-cerebraltissue removal algorithm and analysisrdquoNeuroImage vol 56 no4 pp 1982ndash1992 2011

[50] M Modat G R Ridgway Z A Taylor et al ldquoFast free-form deformation using graphics processing unitsrdquo ComputerMethods and Programs in Biomedicine vol 98 no 3 pp 278ndash284 2010

[51] M Rajchl J Yuan J A White et al ldquoInteractive hierarchical-flow segmentation of scar tissue from late-enhancement cardiacmr imagesrdquo IEEE Transactions on Medical Imaging vol 33 no1 pp 159ndash172 2014

[52] QMahmood A Chodorowski AMehnert andM Persson ldquoAnovel Bayesian approach to adaptive mean shift segmentationof brain imagesrdquo in Proceedings of the 25th IEEE InternationalSymposiumonComputer-BasedMedical Systems (CBMS rsquo12) pp1ndash6 June 2012

[53] A El-Baz M Nitzken A Alansary A Soliman and MCasanova ldquoBrain Segmentation Method for Young Childrenand Adultsrdquo US Provisional Patent Application 13100

[54] A Alansary A Soliman M Nitzken et al ldquoAn integratedgeometrical and stochastic approach for accurate infant brainextractionrdquo in Proceedings of the IEEE International Conferenceon Image Processing (ICIP rsquo14) pp 3542ndash3546 October 2014

[55] V Popescu M Battaglini W S Hoogstrate et al ldquoOptimizingparameter choice for FSL-Brain Extraction Tool (BET) on 3DT1 images in multiple sclerosisrdquo NeuroImage vol 61 no 4 pp1484ndash1494 2012

[56] J L Whitwell W R Crum H C Watt and N C FoxldquoNormalization of cerebral volumes by use of intracranial vol-ume implications for longitudinal quantitative MR imagingrdquoAmerican Journal of Neuroradiology vol 22 no 8 pp 1483ndash1489 2001

[57] S M Smith ldquoFast robust automated brain extractionrdquo HumanBrain Mapping vol 17 no 3 pp 143ndash155 2002

[58] D W Shattuck G Prasad M Mirza K L Narr and A WToga ldquoOnline resource for validation of brain segmentationmethodsrdquo NeuroImage vol 45 no 2 pp 431ndash439 2009

[59] I Isgum M J N L Benders B Avants et al ldquoEvaluation ofautomatic neonatal brain segmentation algorithms the Neo-BrainS12 challengerdquo Medical Image Analysis vol 20 no 1 pp135ndash151 2015

[60] P Moeskops M J N L Benders S M Chita et al ldquoAutomaticsegmentation of MR brain images of preterm infants usingsupervised classificationrdquo NeuroImage 2015

[61] P Moeskops M A Viergever M J N L Benders and IIsgum ldquoEvaluation of an automatic brain segmentationmethoddeveloped for neonates on adult MR brain imagesrdquo in MedicalImaging Image Processing vol 9413 of Proceedings of SPIEInternational Society for Optics and Photonics Orlando FlaUSA February 2015

[62] L Wang Y Gao F Shi et al ldquoLINKS learning-based multi-source IntegratioN frameworK for Segmentation of infant brainimagesrdquo NeuroImage vol 108 pp 160ndash172 2015

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience 15

[17] M B Cuadra L Cammoun T Butz O Cuisenaire and J-P Thiran ldquoComparison and validation of tissue modelizationand statistical classification methods in T1-weighted MR brainimagesrdquo IEEE Transactions on Medical Imaging vol 24 no 12pp 1548ndash1565 2005

[18] F Klauschen A Goldman V Barra A Meyer-Lindenbergand A Lundervold ldquoEvaluation of automated brain MR imagesegmentation and volumetrymethodsrdquoHuman BrainMappingvol 30 no 4 pp 1310ndash1327 2009

[19] H Zaidi T Ruest F Schoenahl and M-L Montandon ldquoCom-parative assessment of statistical brain MR image segmentationalgorithms and their impact on partial volume correction inPETrdquo NeuroImage vol 32 no 4 pp 1591ndash1607 2006

[20] J A Frazier S M Hodge J L Breeze et al ldquoDiagnostic and sexeffects on limbic volumes in early-onset bipolar disorder andschizophreniardquo Schizophrenia Bulletin vol 34 no 1 pp 37ndash462008

[21] A Klein and J Tourville ldquo101 labeled brain images and aconsistent human cortical labeling protocolrdquo Frontiers in Neu-roscience vol 6 article 171 2012

[22] B Van Ginneken T Heimann and M Styner ldquo3D segmen-tation in the clinic a grand challengerdquo in Proceedings of the10th International Conference on Medical Image Computingand Computer Assisted Intervention (MICCAI rsquo07) pp 7ndash15Brisbane Australia October-November 2007

[23] A M Dale B Fischl and M I Sereno ldquoCortical surface-based analysis I Segmentation and surface reconstructionrdquoNeuroImage vol 9 no 2 pp 179ndash194 1999

[24] B Fischl M I Sereno and A M Dale ldquoCortical surface-basedanalysis II inflation flattening and a surface-based coordinatesystemrdquo NeuroImage vol 9 no 2 pp 195ndash207 1999

[25] M Jenkinson C F Beckmann T E J BehrensMWWoolrichand S M Smith ldquoFSLrdquoNeuroImage vol 62 no 2 pp 782ndash7902012

[26] J Ashburner and K J Friston ldquoUnified segmentationrdquo Neu-roImage vol 26 no 3 pp 839ndash851 2005

[27] Y D Reijmer A Leemans M Brundel L J Kappelle and GJ Biessels ldquoDisruption of the cerebral white matter network isrelated to slowing of information processing speed in patientswith type 2 diabetesrdquoDiabetes vol 62 no 6 pp 2112ndash2115 2013

[28] S Klein M Staring K Murphy M A Viergever and J P WPluim ldquoElastix a toolbox for intensity-based medical imageregistrationrdquo IEEE Transactions onMedical Imaging vol 29 no1 pp 196ndash205 2010

[29] K J Friston J T Ashburner S J Kiebel T E Nichols and WD Penny Eds Statistical Parametric Mapping The Analysis ofFunctional Brain Images Academic Press 2011

[30] F Ritter T Boskamp A Homeyer et al ldquoMedical imageanalysis a visual approachrdquo IEEE Pulse vol 2 no 6 pp 60ndash702011

[31] L R Dice ldquoMeasures of the amount of ecologic associationbetween speciesrdquo Ecology vol 26 no 3 pp 297ndash302 1945

[32] D P Huttenlocher G A Klanderman and W J RucklidgeldquoComparing images using the Hausdorff distancerdquo IEEE Trans-actions on Pattern Analysis and Machine Intelligence vol 15 no9 pp 850ndash863 1993

[33] A Alansary A Soliman F Khalifa et al ldquoMAP-based frame-work for segmentation of MR brain images based on visualappearance and prior shaperdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal September 2013

[34] A Jog S Roy J L Prince and A Carass ldquoMR brain segmenta-tion using decision treesrdquo in Proceedings of the MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal Nagoya Japan September 2013

[35] R Katyal S Paneri and M Kuse ldquoGaussian intensity modelwith neighborhood cues for fluid-tissue categorization ofmulti-sequence MR brain imagesrdquo in Proceedings of the MICCAIGrand Challenge on MR Brain Image Segmentation (MRBrainSrsquo13) The Midas Journal Nagoya Japan September 2013

[36] QMahmoodMAlipoor A Chodorowski AMehnert andMPersson ldquoMultimodal MR brain segmentation using Bayesian-based adaptive mean-shift (BAMS)rdquo in Proceedings of theMICCAI WorkshopsmdashThe MICCAI Grand Challenge on MRBrain Image Segmentation (MRBrainS rsquo13) The Midas JournalSeptember 2013

[37] A van Opbroek F van der Lijn andM de Bruijne ldquoAutomatedbrain-tissue segmentation by multi-feature SVM classificationrdquoin Proceedings of the MICCAI WorkshopsmdashThe MICCAI GrandChallenge onMRBrain Image Segmentation (MRBrainS rsquo13)TheMidas Journal September 2013

[38] S Pereira J Festa J A Mariz N Sousa and C A SilvaldquoAutomatic brain tissue segmentation of multi-sequence MRimages using random decision forestsrdquo in Proceedings of theMICCAI Grand Challenge on MR Brain Image Segmentation(MRBrainS rsquo13) The Midas Journal Nagoya Japan September2013

[39] MRajchl J Baxter J Yuan T Peters andAKhan ldquoMulti-atlas-based segmentation with hierarchical max-flowrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[40] D Sarikaya L Zhao and J J Corso ldquoMulti-atlas brain MRIsegmentation with multiway cutrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[41] H Vrooman F van der Lijn and W Niessen ldquoAuto-kNNbrain tissue segmentation using automatically trained k-nearest-neighbor classificationrdquo in Proceedings of the MICCAIWorkshopsmdashTheMICCAI Grand Challenge on MR Brain ImageSegmentation (MRBrainS rsquo13) The Midas Journal September2013

[42] S Vyas P Burlina D Kleissas and R Mukherjee ldquoAutomatedwalks using machine learning for segmentationrdquo in Proceedingsof the MICCAI Grand Challenge on MR Brain Image Seg-mentation (MRBrainS rsquo13) The Midas Journal Nagoya JapanSeptember 2013

[43] C Wang and O Smedby ldquoFully automatic brain segmentationusing model-guided level sets and skeleton-based modelsrdquo inProceedings of theMICCAI Grand Challenge onMRBrain ImageSegmentation (MRBrainS rsquo13) The Midas Journal NagoyaJapan September 2013

[44] C Cortes and V Vapnik ldquoSupport-vector networksrdquo MachineLearning vol 20 no 3 pp 273ndash297 1995

[45] H A Vrooman C A Cocosco F van der Lijn et al ldquoMulti-spectral brain tissue segmentation using automatically trainedk-Nearest-Neighbor classificationrdquo NeuroImage vol 37 no 1pp 71ndash81 2007

[46] C Wang H Frimmel and O Smedby ldquoFast level-set basedimage segmentation using coherent propagationrdquo MedicalPhysics vol 41 no 7 Article ID 073501 2014

16 Computational Intelligence and Neuroscience

[47] A Jog S Roy A Carass and J L Prince ldquoMagnetic resonanceimage synthesis through patch regressionrdquo in Proceedings of theIEEE 10th International Symposium onBiomedical Imaging (ISBIrsquo13) pp 350ndash353 April 2013

[48] J G Sied A P Zijdenbos and A C Evans ldquoA non-parametricmethod for automatic correction of intensity non-uniformity inMRI datardquo IEEE Transactions on Medical Imaging vol 17 no 1pp 87ndash97 1998

[49] A Carass J Cuzzocreo M B Wheeler P-L Bazin S MResnick and J L Prince ldquoSimple paradigm for extra-cerebraltissue removal algorithm and analysisrdquoNeuroImage vol 56 no4 pp 1982ndash1992 2011

[50] M Modat G R Ridgway Z A Taylor et al ldquoFast free-form deformation using graphics processing unitsrdquo ComputerMethods and Programs in Biomedicine vol 98 no 3 pp 278ndash284 2010

[51] M Rajchl J Yuan J A White et al ldquoInteractive hierarchical-flow segmentation of scar tissue from late-enhancement cardiacmr imagesrdquo IEEE Transactions on Medical Imaging vol 33 no1 pp 159ndash172 2014

[52] QMahmood A Chodorowski AMehnert andM Persson ldquoAnovel Bayesian approach to adaptive mean shift segmentationof brain imagesrdquo in Proceedings of the 25th IEEE InternationalSymposiumonComputer-BasedMedical Systems (CBMS rsquo12) pp1ndash6 June 2012

[53] A El-Baz M Nitzken A Alansary A Soliman and MCasanova ldquoBrain Segmentation Method for Young Childrenand Adultsrdquo US Provisional Patent Application 13100

[54] A Alansary A Soliman M Nitzken et al ldquoAn integratedgeometrical and stochastic approach for accurate infant brainextractionrdquo in Proceedings of the IEEE International Conferenceon Image Processing (ICIP rsquo14) pp 3542ndash3546 October 2014

[55] V Popescu M Battaglini W S Hoogstrate et al ldquoOptimizingparameter choice for FSL-Brain Extraction Tool (BET) on 3DT1 images in multiple sclerosisrdquo NeuroImage vol 61 no 4 pp1484ndash1494 2012

[56] J L Whitwell W R Crum H C Watt and N C FoxldquoNormalization of cerebral volumes by use of intracranial vol-ume implications for longitudinal quantitative MR imagingrdquoAmerican Journal of Neuroradiology vol 22 no 8 pp 1483ndash1489 2001

[57] S M Smith ldquoFast robust automated brain extractionrdquo HumanBrain Mapping vol 17 no 3 pp 143ndash155 2002

[58] D W Shattuck G Prasad M Mirza K L Narr and A WToga ldquoOnline resource for validation of brain segmentationmethodsrdquo NeuroImage vol 45 no 2 pp 431ndash439 2009

[59] I Isgum M J N L Benders B Avants et al ldquoEvaluation ofautomatic neonatal brain segmentation algorithms the Neo-BrainS12 challengerdquo Medical Image Analysis vol 20 no 1 pp135ndash151 2015

[60] P Moeskops M J N L Benders S M Chita et al ldquoAutomaticsegmentation of MR brain images of preterm infants usingsupervised classificationrdquo NeuroImage 2015

[61] P Moeskops M A Viergever M J N L Benders and IIsgum ldquoEvaluation of an automatic brain segmentationmethoddeveloped for neonates on adult MR brain imagesrdquo in MedicalImaging Image Processing vol 9413 of Proceedings of SPIEInternational Society for Optics and Photonics Orlando FlaUSA February 2015

[62] L Wang Y Gao F Shi et al ldquoLINKS learning-based multi-source IntegratioN frameworK for Segmentation of infant brainimagesrdquo NeuroImage vol 108 pp 160ndash172 2015

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

16 Computational Intelligence and Neuroscience

[47] A Jog S Roy A Carass and J L Prince ldquoMagnetic resonanceimage synthesis through patch regressionrdquo in Proceedings of theIEEE 10th International Symposium onBiomedical Imaging (ISBIrsquo13) pp 350ndash353 April 2013

[48] J G Sied A P Zijdenbos and A C Evans ldquoA non-parametricmethod for automatic correction of intensity non-uniformity inMRI datardquo IEEE Transactions on Medical Imaging vol 17 no 1pp 87ndash97 1998

[49] A Carass J Cuzzocreo M B Wheeler P-L Bazin S MResnick and J L Prince ldquoSimple paradigm for extra-cerebraltissue removal algorithm and analysisrdquoNeuroImage vol 56 no4 pp 1982ndash1992 2011

[50] M Modat G R Ridgway Z A Taylor et al ldquoFast free-form deformation using graphics processing unitsrdquo ComputerMethods and Programs in Biomedicine vol 98 no 3 pp 278ndash284 2010

[51] M Rajchl J Yuan J A White et al ldquoInteractive hierarchical-flow segmentation of scar tissue from late-enhancement cardiacmr imagesrdquo IEEE Transactions on Medical Imaging vol 33 no1 pp 159ndash172 2014

[52] QMahmood A Chodorowski AMehnert andM Persson ldquoAnovel Bayesian approach to adaptive mean shift segmentationof brain imagesrdquo in Proceedings of the 25th IEEE InternationalSymposiumonComputer-BasedMedical Systems (CBMS rsquo12) pp1ndash6 June 2012

[53] A El-Baz M Nitzken A Alansary A Soliman and MCasanova ldquoBrain Segmentation Method for Young Childrenand Adultsrdquo US Provisional Patent Application 13100

[54] A Alansary A Soliman M Nitzken et al ldquoAn integratedgeometrical and stochastic approach for accurate infant brainextractionrdquo in Proceedings of the IEEE International Conferenceon Image Processing (ICIP rsquo14) pp 3542ndash3546 October 2014

[55] V Popescu M Battaglini W S Hoogstrate et al ldquoOptimizingparameter choice for FSL-Brain Extraction Tool (BET) on 3DT1 images in multiple sclerosisrdquo NeuroImage vol 61 no 4 pp1484ndash1494 2012

[56] J L Whitwell W R Crum H C Watt and N C FoxldquoNormalization of cerebral volumes by use of intracranial vol-ume implications for longitudinal quantitative MR imagingrdquoAmerican Journal of Neuroradiology vol 22 no 8 pp 1483ndash1489 2001

[57] S M Smith ldquoFast robust automated brain extractionrdquo HumanBrain Mapping vol 17 no 3 pp 143ndash155 2002

[58] D W Shattuck G Prasad M Mirza K L Narr and A WToga ldquoOnline resource for validation of brain segmentationmethodsrdquo NeuroImage vol 45 no 2 pp 431ndash439 2009

[59] I Isgum M J N L Benders B Avants et al ldquoEvaluation ofautomatic neonatal brain segmentation algorithms the Neo-BrainS12 challengerdquo Medical Image Analysis vol 20 no 1 pp135ndash151 2015

[60] P Moeskops M J N L Benders S M Chita et al ldquoAutomaticsegmentation of MR brain images of preterm infants usingsupervised classificationrdquo NeuroImage 2015

[61] P Moeskops M A Viergever M J N L Benders and IIsgum ldquoEvaluation of an automatic brain segmentationmethoddeveloped for neonates on adult MR brain imagesrdquo in MedicalImaging Image Processing vol 9413 of Proceedings of SPIEInternational Society for Optics and Photonics Orlando FlaUSA February 2015

[62] L Wang Y Gao F Shi et al ldquoLINKS learning-based multi-source IntegratioN frameworK for Segmentation of infant brainimagesrdquo NeuroImage vol 108 pp 160ndash172 2015

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014