Reproducibility question: unable to match reported BRCA macro F1 (0.82 vs 0.74)

[tstilwell91]

Hello,

First, thank you for your work on BioLM-Net and for making the code and paper available. We’ve been exploring your approach and found the ideas and results very interesting, especially for downstream biological classification tasks.

We are currently trying to reproduce the BRCA results reported in the paper (macro F1 = 0.82), but have not been able to reach that level of performance. Using the provided repository and following the documented setup as closely as possible, our best result so far is around 0.74 macro F1.

We wanted to ask if there are any additional details that might help clarify this gap:

• Were specific hyperparameters used for the BRCA experiment that are not fully reflected in the repo defaults?
• Were there any preprocessing steps, data splits, or filtering choices that differ from what is currently implemented?
• Are there any external scripts, internal utilities, or experimental settings used to produce the reported results that are not included in the repository?

We’d really appreciate any pointers you can share to help us better align with the reported results.

Thanks again for your contribution to the field.

Best,
Terry Stilwell

[jubairmalik]

Sorry for the late reply. We somehow missed your comment.

There are two versions of BioLMNET in the repository: the regular version and the weighted_loss version. We used the weighted_loss version to obtain the previously reported Macro F1 score of 0.82 ± 0.0281.

1. Hyperparameters: The default hyperparameters in the repository are not applicable to all datasets. For the BRCA dataset, we used the following hyperparameter grid:

params_grid = {
    'num_ex_neuron': [0],
    'layer_one': [64, 128],
    'activation_function_one': ['relu', 'sigmoid', 'tanh'],
    'activation_function_two': ['relu', 'sigmoid', 'tanh'],
    'activation_function_three': ['relu', 'sigmoid', 'tanh'],
    'dropout_p': [0.3],
    'learning_rate': [0.01],
    'l2_reg': [0.01],
    'epoch': [200],
    'batch_size': [16, 32],
    'optimizer': ['sgd', 'adam']
}

We also reran the experiment using these settings and obtained the following results:

• Test Macro F1: 0.79 ± 0.0400
• Test Precision: 0.81 ± 0.0486
• Test Recall: 0.82 ± 0.0285
• Test Accuracy: 0.81 ± 0.0241

2. Preprocessing and data splitting: The preprocessing steps, data splits, and filtering choices were kept the same as those reported in the original paper. As mentioned in Section 2.1 (Dataset), the processed TCGA-BRCA and ROSMAP datasets were obtained from Wang et al.

Previously, we provided our processed BRCA data to keep the code consistent with the other datasets. We have now uploaded the processed data from the MOGONET paper to Dataset/BRCA/MOGONET. To reproduce the BRCA results, we recommend using the data in this directory.

Please let us know if you have any other questions or if anything needs further clarification.