Feasibility of using Crested for finetuning on MPRA data

[RaymondPreston]

Description of feature

Hi!

**Context:**I trained a model on bulk ATAC-seq data where my lab previously isolated 16 different subclones representing a continuous trajectory from a low to high metastatic cell state (originating from two parental cell lines of KPC). I did base model training on all peaks, and then fine-tuned my model on the subset of the peaks that are DESeq2 differentially accessible peaks in the met-high vs. met-low subclones. I've had to set up some custom optimizer functions for my use case, but I've been pretty impressed by their in silico performance.

I am currently planning an MPRA on the synthetic enhancers generated from this model to further improve its performance via fine-tuning. My MPRA output will be in the form of RNA/DNA read ratios (log2FC) per cell type. Because this cell state shift is very subtle, I anticipate needing an iterative process of fine-tuning off this MPRA data. My ultimate goal is to generate cell-state-specific reporter constructs.

My question is: Does CREsted natively support fine-tuning on this specific data type (The normalized activity scores rather than accessibility peaks)? To help bridge the gap between chromatin accessibility and transcriptional activity, I intend to include a subset of endogenous regulatory sequences that were present in the original ATAC-seq training data to act as anchors. Any guidance on how to best format this for fine-tuning within the CREsted architecture would be awesome!

Thanks for your time and the great tool!

[casblaauw]

Hey! Glad you're enjoying using CREsted so far!

While we haven't done any MPRA finetuning ourselves, it should definitely be possible. I see no reason why this wouldn't work from your data - we've previously done similar things like first predicting binary labels, then finetuning on scalar values. It's something we'd also be interested in - we've been scoring sequences from MPRA runs with the model, but haven't been finetuning further on those results. I'd be happy to help you get this set up, I think it would also be a great CREsted showcase!

At a practical level, there's two aspects to solve, at a quick glance:

• Adjusting the model to predict differently-ranged values
• Training on non-genomic sequences

The first thing you'd want to do is adjust the final layer, since it by default uses a softplus activation function, meaning it can only predict from 0 to infinity, which of course does not fit your new data. The process for that is pretty simple: get the last layer before the layer you want to replace (where you want to 'intercept' in the model), take that layer's outputs and add a replacement layer on top, and then create a new model with that. This is some pseudocode I currently can't test since our cluster is down, so bear with me:

# Get the name of the last layer before what we call the head layer (the final linear layer predicting your # of cell types)
print(model.summary())

# Get the placeholder output of that layer - assuming you're using the dilatedcnn architecture, this'll be a GlobalAveragePooling1D layer
prehead_output = model.get_layer('GlobalAveragePooling1D').output

# Add a new replacement layer and attach it to the prehead output
# Presumably n_outputs is 1 for your MPRA data
new_output = keras.layers.Dense(n_outputs)(prehead_output)

# Create your new model, ready for further fine-tuning
new_model = keras.Model(inputs = model.inputs, outputs = new_output)

I think for log2(RNA/DNA) data, a raw dense layer without an activation function should be fine? I had a look at how the model used in https://www.nature.com/articles/s41586-024-08430-9 does it, and they also use direct outputs from a linear layer.

For the finetuning process itself, it might be worth first freezing the trunk (non-head part) and re-training the new head on your MPRA data, before unfreezing the entire part, so that the new head has time to converge on something useful before it starts screwing with the weights of the entire carefully trained model. For inspiration, there's code that does something like that in crested.tl.Crested.transferlearn() here, but I'm not sure if anyone has used this recently, so it might need some updates.

For training on non-genomic sequences, CREsted currently doesn't have a built-in method. You could make a custom fake 'genome' fasta with your sequences, but there's an easier solution: I have a private branch with a dataloader rework built with this exact case in mind, so I'll try to get that public somewhere this week.

Hope this helps, and feel free to ask if anything's unclear!