PYTHON BENCHMARKING FRAMEWORK
GenoBench: Python Benchmarking Framework for Genomic Foundation Models
Project Overview
GenoBench is an open-source benchmarking framework published on PyPI (v1.0.0) designed to systematically evaluate and profile Genomic Foundation Models (GFMs).
The tool currently supports models like HyenaDNA-tiny and DNABERT-2, enabling researchers to measure both genomic task accuracy and hardware-level performance metrics (such as memory footprint, token throughput, and latency).
- PyPI Package: GenoBench on PyPI (Note: Placeholder link for demonstration)
- GitHub Repository: M-Ali-Alam/GenoBench (Note: Placeholder link for demonstration)
Inspiration & Research Origins
GenoBench grew out of a master’s thesis research direction exploring hardware-aware inference of linear-time genomic language models.
In genomics, sequence lengths can scale to hundreds of thousands of base pairs. Standard Transformer models scale quadratically ($O(N^2)$) in time and memory complexity relative to sequence length, making them extremely expensive.
This research profiles and compares alternate architectures:
- State Space Models (SSMs) & Mamba: Linear-time architectures designed to capture long-range dependencies efficiently.
- HyenaDNA: Sub-quadratic convolutions designed specifically for long genomic contexts.
- Optimized Transformers (e.g., DNABERT-2): Utilizing flash attention and specialized tokenization to mitigate quadratic overhead.
Key Features
- Unified Model Loader: Load different GFM architectures (HyenaDNA, DNABERT, etc.) with a single line of code.
- Accuracy Profiling: Out-of-the-box support for genomics tasks (e.g., promoter detection, splice site classification, enhancer identification).
- Hardware-Aware Metrics: Real-time logging of GPU/CPU memory usage, latency, and tokens-per-second during both inference and fine-tuning.
- Extensible CLI: Run benchmarks from the command line and export reports in CSV, JSON, or markdown formats.
# Example command to benchmark DNABERT-2 on CUDA
genobench run --model dnabert-2 --task promoter-detection --device cuda --export report.json
How to Get Started
You can install GenoBench directly from PyPI:
pip install genobench
And run a quick evaluation test:
import genobench
# Benchmark HyenaDNA-tiny latency on CUDA
metrics = genobench.benchmark(
model_name="hyenadna-tiny",
task="enhancer-classification",
device="cuda"
)
print(f"Throughput: {metrics.tokens_per_sec:.2f} tokens/second")
print(f"Peak Memory: {metrics.peak_memory_mb:.2f} MB")