Everything matters...also in machine learning: some pointers from ReproNim¶
Peer Herholz (he/him)
Research Assistant Professor - Northwestern University
Research affiliate - NeuroDataScience lab at MNI/McGill,
UNIQUE,
Max-Planck-Institute for Human Cognitive and Brain Sciences
Member - BIDS, ReproNim, Brainhack, Neuromod, OHBM SEA-SIG
@peerherholz
Interactive version:
Let's imagine the following scenario:
Your PI tells you to run some machine learning analyses on your data (because buzz words and we all need top tier publications and that sweet grant money). Specifically, you should use resting state connectivity data to predict the age of participants (sounds familiar, eh?). So you go ahead, gather some data, apply a random forest (a form of decision tree) and ...
Whoa, let's go back a few steps and actually briefly define what we're talking about here, starting with machine learning.
and also its components & steps:
Now, let's actually check how this looks like. At first we get the data:
import urllib.request
url = 'https://www.dropbox.com/s/v48f8pjfw4u2bxi/MAIN_BASC064_subsamp_features.npz?dl=1'
urllib.request.urlretrieve(url, 'MAIN2019_BASC064_subsamp_features.npz')
and then inspect it:
import numpy as np
data = np.load('MAIN2019_BASC064_subsamp_features.npz')['a']
data.shape
We will also visualize it to better grasp what's going on:
import plotly.express as px
from IPython.display import display, HTML
from plotly.offline import init_notebook_mode, plot
fig = px.imshow(data, labels=dict(x="features (whole brain connectome connections)", y="participants"),
height=800, aspect='None')
fig.update(layout_coloraxis_showscale=False)
init_notebook_mode(connected=True)
fig.show()
#plot(fig, filename = 'input_data.html')
#display(HTML('input_data.html'))
Beside the input data we also need our labels:
url = 'https://www.dropbox.com/s/ofsqdcukyde4lke/participants.csv?dl=1'
urllib.request.urlretrieve(url, 'participants.csv')
Which we then load and check as well:
import pandas as pd
labels = pd.read_csv('participants.csv')['AgeGroup']
labels.describe()
For a better intuition, we’re going to also visualize the labels and their distribution:
fig = px.histogram(labels, marginal='box', template='plotly_white')
fig.update_layout(showlegend=False, width=800, height=600)
init_notebook_mode(connected=True)
fig.show()
#plot(fig, filename = 'labels.html')
#display(HTML('labels.html'))
And we’re ready to create our machine learning analysis pipeline using scikit-learn within we will scale our input data, train a Random Forest and test its predictive performance. We import the required functions and classes:
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import GroupShuffleSplit, cross_validate, cross_val_score
from sklearn.metrics import accuracy_score
and setup an scikit-learn pipeline:
pipe = make_pipeline(
StandardScaler(),
RandomForestClassifier()
)
That's all we need to run the analysis, computing accuracy and mean absolute error:
acc_val_rf = cross_validate(pipe, data, pd.Categorical(labels).codes, cv=10, return_estimator =True)
acc_rf = cross_val_score(pipe, data, pd.Categorical(labels).codes, cv=10)
mae_rf = cross_val_score(pipe, data, pd.Categorical(labels).codes, cv=10,
scoring='neg_mean_absolute_error')
which we can then inspect for each CV fold:
for i in range(10):
print(
'Fold {} -- Acc = {}, MAE = {}'.format(i, np.round(acc_rf[i], 3), np.round(-mae_rf[i], 3))
)
and overall:
print('Accuracy = {}, MAE = {}, Chance = {}'.format(np.round(np.mean(acc_rf), 3),
np.round(np.mean(-mae_rf), 3),
np.round(1/len(labels.unique()), 3)))
That's a pretty good performance, eh? The amazing power of machine learning! But there's more: you also try out an ANN to see if it's providing even better predictions.
That's as easy as the "basic machine learning pipeline". Just import the respective functions and classes:
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
Define a simple ANN with 4 layers:
model = keras.Sequential()
model.add(layers.Input(shape=data[1].shape))
model.add(layers.Dense(100, activation="relu", kernel_initializer='he_normal', bias_initializer='zeros'))
model.add(layers.BatchNormalization())
model.add(layers.Dropout(0.5))
model.add(layers.Dense(50, activation="relu"))
model.add(layers.BatchNormalization())
model.add(layers.Dropout(0.5))
model.add(layers.Dense(25, activation="relu"))
model.add(layers.BatchNormalization())
model.add(layers.Dropout(0.5))
model.add(layers.Dense(len(labels.unique()), activation='softmax'))
model.compile(loss='sparse_categorical_crossentropy',
optimizer='adam',
metrics=['accuracy'])
Split the data into train and test again:
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(data, pd.Categorical(labels).codes, test_size=0.2, shuffle=True)
and train your model:
%time fit = model.fit(X_train, y_train, epochs=300, batch_size=20, validation_split=0.2)
That's it! After some quick exploration of the learning success:
import plotly.graph_objects as go
epoch = np.arange(300) + 1
fig = go.Figure()
# Add traces
fig.add_trace(go.Scatter(x=epoch, y=fit.history['accuracy'],
mode='lines+markers',
name='training set'))
fig.add_trace(go.Scatter(x=epoch, y=fit.history['val_accuracy'],
mode='lines+markers',
name='validation set'))
fig.update_layout(title="Accuracy in training and validation set",
template='plotly_white')
fig.update_xaxes(title_text='Epoch')
fig.update_yaxes(title_text='Accuracy')
fig.show()
#plot(fig, filename = 'acc_eyes.html')
#display(HTML('acc_eyes.html'))
you evaluate it on the test set:
score_ann, acc_ann = model.evaluate(X_test, y_test,
batch_size=2)
print('Test score:', score_ann)
print('Test accuracy:', acc_ann)
Amazing, you did, everything is good...or is it?
Your PI is so happy about this fantastic results that now everyone in the lab should run machine learning analyses.
Thus you are ordered to give everyone on the lab your analysis script and a few colleagues try out immediately using the exact same version of the script and data. (I actually asked 6 real people to run it.)
However, all of them get different results...
for the random forest:
Accuracy: 0.52, 0.53, 0.55, 0.58, 0.55
MAE: 1.11, 1.10, 1.12, 1.05, 1.04
and the deep learning analyses, i.e. the ANN:
Test score: 1.55, 1.44, 1.65, 1.54, 0.83
Test accuracy: 0.55, 0.45, 0.55, 0.55, 0.74
Shocked you rerun your old analyses and it gets worse: your results also differ from the previous run.
print('previous Accuracy = {}, previous MAE = {}, previous Chance = {}'.format(np.round(np.mean(acc_rf), 3),
np.round(np.mean(-mae_rf), 3),
np.round(1/len(labels.unique()), 3)))
acc = cross_val_score(pipe, data, pd.Categorical(labels).codes, cv=10)
mae = cross_val_score(pipe, data, pd.Categorical(labels).codes, cv=10,
scoring='neg_mean_absolute_error')
print('Accuracy = {}, MAE = {}, Chance = {}'.format(np.round(np.mean(acc), 3),
np.round(np.mean(-mae), 3),
np.round(1/len(labels.unique()), 3)))
# score, acc = model.evaluate(X_test, y_test,
# batch_size=2)
print('previous Test score:', score_ann)
print('previous Test accuracy:', acc_ann)
fit = model.fit(X_train, y_train, epochs=300, batch_size=20, validation_split=0.2, verbose=0)
score, acc = model.evaluate(X_test, y_test,
batch_size=2)
print('Test score:', score)
print('Test accuracy:', acc)
What is going on...
via https://c.tenor.com/oKay8GcV660AAAAC/ted-dancon-evil-laugh.gif </small></small></small></small>
The inconvenient truth is: as every other analyses you might run within the field of neuroimaging (or other any research field), there is an substantial amount of factors that contribute to the reproducibility of your machine learning analyses. Even more so, a lot of the problems are actually elevated due to the complex nature of the analyses.
But why care about reproducibility in machine learning at all and how can some of the underlying problems be addressed? Let's have a look...
Why care about reproducibility in machine learning?¶
We all know the reproducibility crisis in neuroimaging...
but we also know that neuroimaging is by far no exception to the rule as many other, basically all, research fields have comparable problems.
This also includes machine learning...
adapted from [Martina Vilas](https://doi.org/10.5281/zenodo.4740053)</small>
Besides obviously being a major shortcoming as indicated by this quote from Popper (The Logic of Scientific Discovery),
Non-reproducible single occurrences are of no significance to science.
adapted from [Suneeta Mall](https://suneeta-mall.github.io/2019/12/21/Reproducible-ml-research-n-industry.html)</small>
what are crucial aspects when talking about reproducibility in machine learning (with a focus on neuroimaging)?
adapted from [Suneeta Mall](https://suneeta-mall.github.io/talks/KubeCon_US_2019)</small>
Understanding, Explaining, Debugging, and Reverse Engineering¶
- reproducibility helps with
understanding,explaining, anddebugging& is crucial toreverse engineering
Why is this important?
machine learningis very difficult tounderstand,explainanddebug- unreproducible outcomes makes it increasingly harder
- if outcomes are unreproducible & we don't understand what's going in the first place how can we reverse engineer anything?
adapted from [Suneeta Mall](https://suneeta-mall.github.io/2019/12/21/Reproducible-ml-research-n-industry.html)</small>
Our example:
- we want to know why our
modelprovided us with the obtainedaccuracy - we want to know what our
model"learned" from our data - we want to know if certain
featuresare more important than others
Our problem(s):
- we get a different
accuracyevery time we run ourmodel - we can't be sure what our
modellearned from the data - we can't be sure that certain
featuresare more important than others
adapted from [Suneeta Mall](https://suneeta-mall.github.io/2019/12/21/Reproducible-ml-research-n-industry.html)</small>
Correctness¶
- reproducibility helps with correctness via understanding and debugging
Why is this important?
- we want to be sure that the results/effects we observe are actually there or only a fluke
- important for knowledge generation and advancement in science
- more and more researchers utilize
machine learningto makepredictionsabouthumans(e.g.group clustering,treatment/therapy plans,disease propagation, etc.) andunreproducible model predictionscan lead to devastating errors
Our example:
- we want to be sure that our
modelactually "learned" a way toreproducibly predicttheageofparticipantsfrom theirresting state connectome
Our problem(s):
- we can't be sure that our
modelactually "learned" in areproduciblemanner as the outcome varies - we can't "trust" our
model
adapted from [Suneeta Mall](https://suneeta-mall.github.io/2019/12/21/Reproducible-ml-research-n-industry.html)</small>
Credibility¶
- easily
reproducible(machine learning) results are morecredible
Why is this important?
- we want
FAIR,verifiable,reliable,unbiasedðicalanalysesandresults - in the near future a lot of very important decisions that will impact lives will depend on
resultsfrommachine learning analyses - we already saw stunning examples of how easily things go tremendously wrong
adapted from [Suneeta Mall](https://suneeta-mall.github.io/2019/12/21/Reproducible-ml-research-n-industry.html)</small>
Our example:
- we want to rely on our result that
agecan be predict fromresting state connectomesin averifiableandunbiasedway that isethicalandFAIR
Our problem(s):
- none of these requirements are fulfilled if our results are not reproducible
Extensibility¶
reproducibilityhelps withextensibilityof successive steps in amachine learning analyses
Why is this important?
- a given part of a
machine learning analyses(e.g.feature engineering,layersof anANN, etc.) need to bereproducibleso that the respectiveanalyses(or pipeline) can be extended - if results are not
reproduciblesubsequent steps like differentpost-processingoptions andaugmentationare not feasible/possible
adapted from [Suneeta Mall](https://suneeta-mall.github.io/2019/12/21/Reproducible-ml-research-n-industry.html)</small>
Our example:
- we want to extract
featuresimportant for thepredictionofagefrom themodel - we want to
augmentourmodelwith newfeaturesfrom a different modality (e.g.DTI,behavior, etc.)
Our problem(s):
- we can't be sure that the importance we extract for the
featuresis actually reliable - we can't
augmentourmodelbecause it's current state producesunreproducibleresults
Data harvesting¶
reproducibilityhelps withmodel trainingthroughdata generation
Why is this important?
- any type of
machine learning analysesneeds tremendous amounts ofdatato betrained,evaluatedandtestedon - due to lack of
data sharingpractices andstandardization, as well asquality controlthere's actually not enoughdatafor most planned (or even conducted)analyses - this is especially true for certain types of
data(e.g.disorders/diseasewith a low prevalence and/or broad spectra) generation of synthetic dataviamachine learning(e.g.GANs) offers amazing possibilities to address this problem- for this to work out/make sense, the outcomes, i.e.
synthetic data, of themodelneed to bereproducible
adapted from [Suneeta Mall](https://suneeta-mall.github.io/2019/12/21/Reproducible-ml-research-n-industry.html)</small>
Our example:
- if we want to
generatemoredatabased on the one we have (e.g. alldata, importantfeatures, etc.), thegenerationof thisdatashould bereproducible - if we want to use aspects of the
dataforsynthesizing, thederivationof thisdatashould bereproducible
Our problem(s):
- the
model performance, thus alsoderivationofdatais notreproducible
At this point you might think:
via https://c.tenor.com/LOuJtZ-WL3kAAAAC/holy-forking-shirt-shocked.gif</small></small></small>
and you would be right: there are so many things to consider and that can go wrong.
However, we shouldn't give up quite yet and instead have a look at the underlying problems and how we can address them!
Challenges in reproducible machine learning¶
Here's another inconvenient truth: every single aspect involved in a machine learning analyses creates variability and thus entails a major hurdle towards achieving reproducibility.
This not only entails the computational infrastructure one is working with but also the data as well as common practices during the application of these analyses.
adapted from [Suneeta Mall](https://suneeta-mall.github.io/talks/KubeCon_US_2019)</small>
Hardware¶
machine learningalgorithms require intensive computation and thus can take a very long time to run- specialized hardware to address this via e.g.,
parallelism&multiple processing units(CPU),graphics processing units(GPU),tensor processing units(TPU), etc.
adapted from [Suneeta Mall](https://suneeta-mall.github.io/2019/12/21/Reproducible-ml-research-n-industry.html)</small>
- while very helpful, this introduces problems concerning
reproducibilitybased on the underlyingfloating-point computationsand appears independent of thehardware parallelismonCPU, bothintra-ops&inter-ops(within operation/across multiple operations) can produce different outcomes when run repeatedlystream multiprocessing(SEM) units inGPUs, i.e.asynchronous computation, can lead to variable outcomes during repeated runs- changing the respective
architecturescan also changemodel outcomes
adapted from [Suneeta Mall](https://suneeta-mall.github.io/2019/12/21/Reproducible-ml-research-n-industry.html)</small>
Software¶
- most commonly used
frameworksformachine learning analysesactually don't guarantee 100%reproducibility, includingcudnn,pytorchandtensorflow - there's always the danger of
bugswhich can be hard to find based on high-levelAPIs
adapted from [Suneeta Mall](https://suneeta-mall.github.io/2019/12/21/Reproducible-ml-research-n-industry.html)</small>
- every
computationrequires a complex stack ofsoftwarethat quite often interacts and has cross-dependenciesoperating system,drivers,python,python packages, etc.- versions of all of the above
numerical errors & instabilitiesof thesoftware, e.g. talking aboutlocal minima/maxima&floating-point precision againerrorsmay lead unstablefunctionstowards distinctlocal minima
via https://brilliant.org/wiki/extrema/</small></small>
print(sum([0.001 for _ in range(1000)]))
Algorithm¶
- the characteristics of quite a few
machine learning algorithmsintroduce further problems - the expectation of
randomnesscan lead tounreproducibleoutcomes - complexity of
computationcan lead tonon-deterministicresults
adapted from [Suneeta Mall](https://suneeta-mall.github.io/2019/12/21/Reproducible-ml-research-n-industry.html)</small>
Practices/Process¶
- entailing aspects of both previous points,
softwareandalgorithm,practicesandprocessesinmachine learningalso heavily influence thereproducibilityof results - most prominently this entails
randomness, because it's basically everywhere
random initializationsrandom augmentationsrandom noise introductiondata shuffles- etc.
adapted from [Suneeta Mall](https://suneeta-mall.github.io/2019/12/21/Reproducible-ml-research-n-industry.html)</small>
Here's just a brief example concerning data shuffling in training sets:
for i in range(100):
X_train, X_test, y_train, y_test = train_test_split(data, pd.Categorical(labels).codes, test_size=0.2, shuffle=True)
pd.Series(y_train).plot(kind='hist', stacked=True)
Data¶
- even though commonly ignored:
machine learning analysisneed tremendous amounts ofdata, the buzz word is "big data" - given that
data standardization & managementare still not frequently applied, even in smalldatasets, "big data" poses additional major problems
- when working with any kind of
data, especially "big data" various aspects need to be addresseddata managementdata provenancedata poisoningunder-/over-represented data
adapted from [Suneeta Mall](https://suneeta-mall.github.io/2019/12/21/Reproducible-ml-research-n-industry.html)</small>
- this concerns basically every aspect of
machine learning analyses, e.g.feature engineering,augmentation, etc.
- another important aspect that interacts with all of the above is the limited re-usability of
models - only a small subset of
modelsis used more than once, which is grounded in two main factors (among others)
- absent sharing of
trained modelstraining modelscan take a very long time as previously outlined and additionally consume large amounts of resources- sharing
trained modelsorweightsthereof is necessary to reduce these costs and further validate themodelsin terms ofreproducibility,reliability,robustnessandgeneralization
- everything we do with
machine learningis more or less affected byconcept drift- our
conceptof the world that surrounds us and the things within is constantly changing (e.g.disease/disordersubtypes and progressions, etc.) - we as
biological agentsneed to makeartificial agentsaware of this - in
machine learningthis process is referred to ascontinual learning
- our
adapted from [Suneeta Mall](https://suneeta-mall.github.io/2019/12/21/Reproducible-ml-research-n-industry.html)</small>
At this point you might think:
via [GIPHY](https://media2.giphy.com/media/cRMGqNpvm9XS2gRcpL/giphy.gif)</small></small></small>
But again: there's something we can do to make it at least slightly better. Thus, let's get to it!
Pointers to increase reproducibility in machine learning¶
- given the outlined challenges, we can now think about how we can address the respective problems
- the good thing is: a lot of the work and efforts done to increase
reproducibilityinneuroimagingcan also be utilized to increasereproducibilityinmachine learning
this thus also entails tools and resources provided by
ReproNimand its members, as well as adjacent initiativesin more detail we will have a look at the following challenges and respectively helpful aspects
software:virtualizationofcomputing environmentsusing neurodockeralgorithm/practices/processes: dealing withrandomnessdata:standardization(BIDS), tracking everything (git/github & datalad), sharing
- unfortunately,
ReproNimdoesn't offer tohardwareresources yet, so we have to keep that aside for now
Software: virtualization of computing environments using neurodocker¶
- as mentioned above every
computing environmentis composed of many different aspectsoperating system,drivers,python,python packages- different
versionsthereof - etc.
Obviously, we can't send around our machines via post.
So what can we do to address this?
We can make use of virtualization technologies that aim to:
- isolate the computing environment
- provide a mechanism to encapsulate environments in a self-contained unit that can run anywhere
- reconstructing computing environments
- sharing computing environments
Overall, there are several types of virtualization that are situated across different levels:
software containerslike docker & singularityvirtual machineslike Virtualbox & VMware
As discussed before, machine learning analyses need a lot of computational resources to run in a feasible amount of time. Thus, usually we want to make use of HPCs which excludes virtual machines from the suitable options, as their handling, management and resource allocation don't really scale.
In contrast, both python virtualization and software containers work reasonably well on HPCs (regarding utilization, management and resource allocation).
The creation, combination and management of both is made incredibly easy and even reproducible via neurodocker!
- a
containerized softwareaimed at thereproducible generationof othercontainerized software - supports
python virtualizationusingconda,dockerandsingularity, as well asneuroimaging-specific software
You might wonder: "why do we need both"?
python virtualizationdoes not account for often requiredlibrariesandbinarieson other levels (beyondpython)- for this we need the subsequent level of
virtualization, i.e.software containersthat share thehost system's kerneland (can) include everything fromlibrariesandbinariesupwards
In order to create a software container, here docker, that entails a python environment with all packages in the versions that were used to run the initial analyses, we only need to do the following:
%%bash
docker run kaczmarj/neurodocker:0.6.0 generate docker --base=ubuntu:18.04 \
--pkg-manager=apt \
--install git nano unzip \
--miniconda \
version=latest \
create_env='repronim_ml' \
activate=true \
conda_install="python=3.11.0 numpy=1.26.4 pandas=2.3.0 scikit-learn=1.7.0 seaborn=0.13.2" \
pip_install="tensorflow==2.16.2 datalad[full]==1.2.0" \
--add-to-entrypoint "source activate repronim_ml" \
--entrypoint "/neurodocker/startup.sh python" > Dockerfile
docker build -t peerherholz/repronim_ml:0.2 .
As this takes a while, we can also just grab the generated container from dockerhub:
%%bash
docker pull peerherholz/repronim_ml:0.2
With that, we already have our software container ready to go and can run our machine learning analyses in it:
%%bash
docker images
Here, the software container is set up in a way that it automatically executes/runs python, specifically the version of the conda environment we created. Thus, we can simply provide a python script as an input.
%%bash
docker run -i --rm -v /Users/peerherholz/google_drive/GitHub/repronim_ML/:/data peerherholz/repronim_ml:0.2 /data/code/ml_reproducibility_data.py
Fantastic, now we can run and further test our machine learning analyses in a dedicated computing environment that is isolated and shareable!
Now that a large portion of the software dependencies are addressed, we can continue with the next part: algorithm/practices/processes!
algorithm/practices/processes : dealing with randomness¶
- as discussed above,
randomnessis basically everywhere in the world ofmachine learning - while often expected or even required, if not done "right" it can/will lead to
unreproducibleresults
So what can we do?
Simply put, we need to seed the randomness in our machine learning analyses!
- all
softwarehas the option to set theseedforrandomness, one way or another - where and how the
seedcan be set unfortunately varies betweensoftware packages - often, the
seedneeds to be set at different instances
For example, in our machine learning analyses using a random forest, we didn't seed randomness so far, resulting in different outcomes at every run because e.g. training and test set splits would vary:
from sklearn.model_selection import StratifiedKFold
cv = StratifiedKFold()
pipe = make_pipeline(
StandardScaler(),
RandomForestClassifier()
)
for i in range(10):
acc = cross_val_score(pipe, data, pd.Categorical(labels).codes, cv=cv)
mae = cross_val_score(pipe, data, pd.Categorical(labels).codes, cv=cv,
scoring='neg_mean_absolute_error')
print('Accuracy run {} = {}, MAE run {} = {}, Chance run {} = {}'.format(i, np.round(np.mean(acc), 3),
i, np.round(np.mean(-mae), 3),
i, np.round(1/len(labels.unique()), 3)))
Using the random_state argument in all functions/classes that deal with randomness to set the seed, results in reproducible outcomes at every run:
cv = StratifiedKFold(random_state=42, shuffle=True)
pipe = make_pipeline(
StandardScaler(),
RandomForestClassifier(random_state=42)
)
for i in range(10):
acc = cross_val_score(pipe, data, pd.Categorical(labels).codes, cv=cv)
mae = cross_val_score(pipe, data, pd.Categorical(labels).codes, cv=cv,
scoring='neg_mean_absolute_error')
print('Accuracy run {} = {}, MAE run {} = {}, Chance run {} = {}'.format(i, np.round(np.mean(acc), 3),
i, np.round(np.mean(-mae), 3),
i, np.round(1/len(labels.unique()), 3)))
Regarding our ANN, we would need to set the random_state argument for the train_test_split function to create training and test sets in a reproducible manner:
for i in range(100):
X_train, X_test, y_train, y_test = train_test_split(data, pd.Categorical(labels).codes, test_size=0.2, shuffle=True)
pd.Series(y_train).plot(kind='hist', stacked=True)
for i in range(100):
X_train, X_test, y_train, y_test = train_test_split(data, pd.Categorical(labels).codes,
test_size=0.2, shuffle=True,
random_state=42)
pd.Series(y_train).plot(kind='hist', stacked=True)
Additionally, we would need to set the seed across various parts of our computational environment. Without that, we get different results at every run:
for i in range(10):
model = keras.Sequential()
model.add(layers.Input(shape=data[1].shape))
model.add(layers.Dense(100, activation="relu", kernel_initializer='he_normal', bias_initializer='zeros'))
model.add(layers.BatchNormalization())
model.add(layers.Dropout(0.5))
model.add(layers.Dense(50, activation="relu"))
model.add(layers.BatchNormalization())
model.add(layers.Dropout(0.5))
model.add(layers.Dense(25, activation="relu"))
model.add(layers.BatchNormalization())
model.add(layers.Dropout(0.5))
model.add(layers.Dense(len(labels.unique()), activation='softmax'))
model.compile(loss='sparse_categorical_crossentropy',
optimizer='adam',
metrics=['accuracy'])
fit = model.fit(X_train, y_train, epochs=30, batch_size=20, validation_split=0.2, verbose=0)
score, acc = model.evaluate(X_test, y_test,
batch_size=2, verbose=0)
print('Test score run {}: {} , Test accuracy run {}: {}'.format(i, score, i, acc))
After setting the seed, for example the general random seeds and the tensorflow seed, we get the same output every run:
import os, random
for i in range(10):
# Set all seeds at the beginning, before the loop
os.environ['PYTHONHASHSEED'] = str(42)
random.seed(42)
np.random.seed(42)
tf.random.set_seed(42)
model = keras.Sequential()
model.add(layers.Input(shape=data[1].shape))
model.add(layers.Dense(100, activation="relu", kernel_initializer='he_normal', bias_initializer='zeros'))
model.add(layers.BatchNormalization())
model.add(layers.Dropout(0.5))
model.add(layers.Dense(50, activation="relu"))
model.add(layers.BatchNormalization())
model.add(layers.Dropout(0.5))
model.add(layers.Dense(25, activation="relu"))
model.add(layers.BatchNormalization())
model.add(layers.Dropout(0.5))
model.add(layers.Dense(len(labels.unique()), activation='softmax'))
model.compile(loss='sparse_categorical_crossentropy',
optimizer='adam',
metrics=['accuracy'])
fit = model.fit(X_train, y_train, epochs=30, batch_size=20, validation_split=0.2, verbose=0)
score, acc = model.evaluate(X_test, y_test, batch_size=2, verbose=0)
print('Test score run {}: {} , Test accuracy run {}: {}'.format(i, score, i, acc))
Please note, that this appears to be sufficient for our minimal example but more realistic ANNs would potentially require way more steps to achieve reproducibility, including:
- setting
inter/intra op parallelism
from tensorflow.python.eager import context
context._context = None
context._create_context()
tf.config.threading.set_inter_op_parallelism_threads(1)
tf.config.threading.set_intra_op_parallelism_threads(1)
- address
determinismviaosand/or utilizingpackages, like Framework determinism but starting withtensorflowversion2.8, you can set it directly:
# os.environ['TF_DETERMINISTIC_OPS'] = '1'
# from tfdeterminism import patch
# patch()
tf.config.experimental.enable_op_determinism()
The respective solution will depend on your computational environment, model architecture, data, etc. and might sometimes not even exist (at least within a reasonable implementation).
This ongoing concern itself became a new field of study, with fascinating and very helpful outputs, e.g.
Bouthillier et al. (2021): Accounting for variability in machine learning benchmarks
adapted from [Martina Vilas](https://doi.org/10.5281/zenodo.4740053)</small>
data: standardization (BIDS), tracking everything (git/github & datalad), sharing¶
- as mentioned before,
datais crucial & thus everything adjacent to it is crucial as well standardizationofdata&modelsto facilitatemanagement,QCandFAIR-ness- keeping track of everything using
version control sharingdata&modelsto maximizereproducibility&FAIR-ness, as well as to reducecomputational effort
standardization of data & models¶
standardizingdata&models, as well as their description is tremendously helpful & important concerningreproducibilityand other aspects, independent of thedataat handbig datahowever profits even more from this process as the combination ofhuman/machinereadability andmeta-dataallows to efficiently handledata management,evaluation,QC,augmentation, etc.
One of the best options to tackle this is of course the Brain Imaging Data Structure or BIDS:
Keeping track of everything using version control¶
- as mentioned before:
machine learning analysestend to be very complex and thus a lot can happen along the way - the
analysis code, as well asdata inputandoutputmight change frequently - without keeping track of these frequent changes,
reproducibilityis basically not possible
One commonly know tool to address this is version control, which itself can take different forms, depending on the precise data at hand.
Placing everything one does under version control via these tools & resources can help a great deal to achieve reproducibility, as the respective aspects, e.g. code & data are constantly monitored and changes tracked.
For example, starting with our machine learning analyses python script, what if we apply certain changes and the outcomes vary as a result of it? Using version control, it's "easy" to evaluate what happened and restore previous versions!
We actually adapted our python script so that the analyses are more reproducible. With git, we can see that:
While code version control using git and GitHub is fortunately more and more common, version control of other forms of data, e.g. datasets is not.
For example, what happens when we run multiple machine learning analyses on the same dataset that yield different outcomes? How do we keep track of that?
One up and coming tool that allows to "easily" track changes in all kinds of data, is DataLad.
With it, one can create version controlled datasets of basically any form and monitor every change that appears along the way, including inputs and outputs, files in general, utilized software, etc. .
Applied to our machine learning analyses, this could look as follows:
We create a new dataset:
%%bash
datalad create repronim_ml_showcase
And download the data, installing it as dataset:
%%bash
cd repronim_ml_showcase
datalad download-url \
--archive \
--message "Download dataset" \
'https://www.dropbox.com/s/v48f8pjfw4u2bxi/MAIN_BASC064_subsamp_features.npz?dl=1'
%%bash
cd repronim_ml_showcase
datalad download-url \
--message "Download labels" \
'https://www.dropbox.com/s/ofsqdcukyde4lke/participants.csv?dl=1'
We then can use DataLad's YODA principles to create a directory structure for our project, including folders for data and code:
%%bash
cd repronim_ml_showcase
datalad create -c text2git -c yoda ml_showcase
and subsequently clone the installed datasets into the data folder, indicating that this is our raw data:
%%bash
cd repronim_ml_showcase/ml_showcase
mkdir -p data
datalad clone -d . ../ data/raw
To make things as reproducible as possible, we also add our software container from above to the dataset, so that the computational environment and it's application is also version controlled:
%%bash
cd repronim_ml_showcase/ml_showcase
datalad containers-add repronim-ml-container --url dhub://peerherholz/repronim_ml:0.2
The only thing that's missing is the python script that performs the machine learning analyses, which can simply be added to the dataset as well.
%%bash
cd repronim_ml_showcase/ml_showcase/
cp ../../code/ml_reproducibility.py code/
datalad save -m "add random forest & ANN script" code/ml_reproducibility.py
With that, we already have everything in place to run a (fully) reproducible machine learning analysis, that tracks the datasets, the computational environment, the code and also their interaction via monitoring analyses inputs and outputs:
%%bash
cd repronim_ml_showcase/ml_showcase/
datalad containers-run -n repronim-ml-container \
-m "First run of ML analyses" \
--input 'data/raw/' \
--output 'metrics.json' \
--output 'random_forest.joblib' \
--output 'ANN.h5' \
"code/ml_reproducibility.py"
Not only can we obtain reproducible results via the combination of software containers and seeding:
%%bash
cd repronim_ml_showcase/ml_showcase/
cat metrics.json
but we also get a full log of everything that happened in and to our dataset.
%%bash
cd repronim_ml_showcase/ml_showcase/
git log
This would allow to rerun a specific analyses using the same computational environment and data, as well as track quite a bit of variability introduced by changes!
There are also other tools out there, more tailored towards machine learning. For example, mlflow:
https://mlflow.org/docs/latest/tracking.html
adapted from [Martina Vilas](https://doi.org/10.5281/zenodo.4740053)</small>
https://mlflow.org/docs/latest/projects.html
adapted from [Martina Vilas](https://doi.org/10.5281/zenodo.4740053)</small>
Now that we have addressed a good amount of the introduced challenges to reproducible machine learning, including software, algorithms/practices/processes and data, we already achieved a fair level of reproducibility. However, we can even do more!
sharing data & models¶
- as mentioned before: to maximize
reproducibilityandFAIR-ness,dataandmodels, basically everything involved in theanalysesneed to shared - allows others to re-run
analysesor adapt them, reducescomputational costs - various options to share
dataand different types thereof, both openly and with restricted access - the "right" way depends on the
dataat hand
For example, we can store all things on GitHub, including the software/information re computing environments:
with the computing environment being stored/made available on dockerhub:
the code, i.e. machine learning analyses scripts:
and the pre-trained models:
speaking of which: there are amazing projects out there that bring this to the next level, e.g. nobrainer:
Adjacent to that, it's of course required to share as much about the analyses as possible when including it in a publication, etc.. There are cool & helpful checklists out there:
https://www.cs.mcgill.ca/~jpineau/ReproducibilityChecklist.pdf
adapted from [Martina Vilas](https://doi.org/10.5281/zenodo.4740053)</small>
See? There are many things we can do to make our machine learning analyses more reproducible. Unfortunately, all of this won't guarantee full reproducibility. Thus, always make sure to check things constantly!
However, for now let's briefly summarize what we talked about!
Reproducible machine learning - a summary¶
There are also many other very useful resources out there!
Isdahl & Gunderson (2019)
https://www.cs.mcgill.ca/~ksinha4/practices_for_reproducibility/
adapted from [Martina Vilas](https://doi.org/10.5281/zenodo.4740053)</small>
via [GIPHY](https://media3.giphy.com/media/iJ2cRDeQkcPXZiHh53/giphy.gif)</small></small></small>
Thank you all for your attention! If you have any questions, please don't hesitate to ask or contact me via herholz do peer at gmail dot com or via "social media":
@peerherholz
Make sure to also check the ReproNim website: https://www.repronim.org/