The study plan
The question, the data, the methods to beat, and the pass and fail lines, fixed before modelling.
CodecComputational biology
Does a cell’s history predict what it will become?
Codec is being built to test that.
Codec is in early development. We are looking for research groups who want to find out, on their own data, if a cell’s history predicts its fate.
- Agree the question
- Test on cells the model has not seen
- Share the code and results
Cells that look the same can go on to become different things.
Two cells that look the same today can still become different cell types. A 2020 study of blood-cell development showed this. Most prediction methods look only at a cell as it is now. Part of the answer may sit in the cell’s history.
Before you build a paper, a protocol or a programme on a prediction about cell fate, you need to know it will hold on the next donor, the next batch and the next experiment. If it does not hold, your result will not replicate.
Test every prediction on cells the model has never seen.
Results that do not hold up cost years, and money.
28 billionUS dollars a year: the estimated cost of preclinical research that cannot be reproduced, in the United States alone
Codec is designed to compare against the simple method first, on data the model has never seen. See how, step by step
After the study
What a Codec study is designed to give you.
- A measured answer to your question, with its uncertainty.
- A test of the prediction on donors and batches the model has never seen.
- The code and instructions to re-run every number.
- A result you can publish, whatever the answer.
Before any model is fitted, we write down the test and what would count as failure.
The study plan, with its pass and fail lines set in advance.
Each question gets a pass line and a fail line, written down before any model is fitted. A question passes only if the gain holds on unseen clones and donors and repeats across biological samples. If it does not, the simpler model wins and we say so.
The first step is to reproduce a published method, for example CellRank. Any model that uses cell history must beat it, and the simple methods, to count.
Uncertainty is measured across biological replicates, not across individual cells.
Every number will be traceable to its source.
For each value, the study records the dataset and its version, the software version and the model version, and marks the value as observed or predicted. At the end, your team gets the code and instructions to re-run every number.
What the record holds for each value
Where it came from and how it was produced, so a reviewer, or your own team a year later, can follow it back.
- Dataset and versionRecorded
- Software versionRecorded
- Model versionRecorded
- Observed or predictedMarked
- Licence and checksum of every fileRecorded
How a study works
A study has three stages: define, test, hand over.
- 1
Define
We agree one question, one biological system and the data, yours or public. We write down the method to beat, the size of improvement that would matter, and the result that would count as failure. Nothing is fitted until this is agreed.
- 2
Test
We rebuild the data pipeline from raw files and reproduce a published result first. Then we compare models with and without the cell’s history, on whole clones, donors and batches kept aside from training.
- 3
Hand over
You get the answer, its uncertainty, and the code and instructions to re-run it.
Our promises
What we promise before you start.
- Your scientists lead on the biology. We run the software and the comparisons.
- The pass line and the fail line are agreed in writing before any model is fitted.
- You get the answer whichever way it falls. If history adds nothing in your system, we say so plainly, and you keep the simpler model.
- The work runs in stages. After each one, you see the result and decide if you want to continue.
- Your data stays under your control. Access, handling and permissions are agreed in writing before anything is shared.
- Authorship, publication and ownership of what we build are agreed before the work starts.
- At the end, your team gets the code and instructions to re-run every number.
What you receive
Three things, all yours to keep.
The result report
How each model did on cells, clones and donors it never saw, with uncertainty across biological replicates, including what did not work.
The code
Code, a locked software environment and a record of every dataset (source, licence, checksum), so your team can re-run the study.
How to start
Recommended first step
Your first Codec study
Codec is at an early stage, so the first study is a research collaboration: one biological system, one question, your data or suitable public data. It runs in three stages, each ending in a go or stop decision. Scope, terms and funding are agreed in writing before we start.
- Stage 1Can the data answer the question? The pipeline is rebuilt from raw files and the data checked against the plan.
- Stage 2Can we reproduce the published result on that data? The comparison starts from a result the field already accepts.
- Stage 3Does history improve the prediction, on clones and donors the model never saw? You get the answer, whichever way it falls.
Your own data not ready yet? We can start on a public lineage-tracing dataset for the same system.
Want to publish together? Authorship and publication are agreed in writing before the work starts.
Is a Codec study right for you?
Who it is for
Research groups and biotech R&D teams studying how cells choose their fate, for example in blood-cell development, with lineage-traced single-cell data.
What we need from you
A biological question; lineage-traced single-cell data, or agreement to start on a public dataset; and one scientist who owns the biology.
For each person in the decision
- Principal investigator or head of R&DWhat the study answers and how it will be judged.
- Computational leadThe methods we compare against, how the data is split, how uncertainty is reported.
- Research office or legalData handling, authorship, publication and ownership, agreed up front.
- Finance or grantsStaged work, each stage scoped in writing before it starts.
The data a study needs, and the methods it is compared against.
- Lineage-traced single-cell dataThe core of every study: cells whose clone is known.
- Time coursesMeasurements over time, to capture each cell’s history.
- Public datasets we can start fromLARRY and other public lineage-tracing datasets.
- Methods a study would compare againstCellRank, Dynamo and Waddington-OT. Your tools stay.
Questions we are asked first.
How far along is Codec?
Codec is at an early stage. The first studies will be run with research partners.
What data do we need to start?
Lineage-traced single-cell data. If yours is not ready, a public dataset for the same system.
Can we start before our own data exists?
Yes. We start on a public dataset for the same system and bring your data in when it is ready.
What if history makes no difference?
You get that answer, measured, and the simpler model. That is a publishable result.
Does this replace CellRank or our current tools?
They are what we compare against. Your tools stay.
Who owns the results, and can we publish?
Agreed in writing before work starts.
How is our data handled?
Agreed in writing before anything is shared, with access limited to named people.
How long does it take?
The first study runs in three stages, each ending in a decision to continue or stop. We set the timing in the written scope once data access is agreed.
How is it funded?
As a commissioned study, or as part of a funding application. We agree it by email and set it out in the written scope.
Is it for clinical use?
No. The study is research on cells in a defined experimental system. Diagnosis and treatment decisions are outside it.
How a Codec study works.
A Codec study asks one question in one biological system: does knowing a cell’s history improve the prediction of what it becomes? We start from measured cells, your data or a suitable public lineage-tracing dataset, and record where every file came from. Before any model is fitted, we agree the outcome to predict, the methods to beat, the improvement that would matter and the result that would count as failure.
We then rebuild the processing pipeline and reproduce a published result, so the comparison starts from a result the field already accepts. Next we compare simple models with models that also use the cell’s history, each tested on whole clones, donors and batches the models never saw. We report how well each one scores, how far its probabilities can be trusted, and how certain we are, measured across biological replicates.
Read more: the technical detail
- The model, in words. A cell’s current state, its measured history, and the probability of reaching each fate. The same model is compared with and without the history term.
- How the models are scored. Held-out negative log-likelihood and Brier score; calibration error (a 70% prediction should come true about 70% of the time); and sensitivity on rare fates.
- How the data is split. By clone, donor, batch and time. Whole groups are kept aside, never random cells, so the model cannot see the answer.
- Uncertainty. Measured across biological replicates, not across individual cells.
- The evidence chain. From measured cells to a published-method reproduction, to the comparison, to an independent replication of the effect where the data allows it.
- Prior work the plan builds on. Weinreb et al. 2020 (lineage tracing in blood-cell development) and the published methods named above.
Tell us the question about cell fate you want to test.
Email us about the system, the question and the data you have. We reply with a plan for the first stage of the study.
Sovereign