arrow_back Back to Research

PYTHON BENCHMARKING FRAMEWORK

GenoBench: Python Benchmarking Framework for Genomic Foundation Models

v1.0.0PythonPyPIPyTorchHuggingFaceGenomicsMambaTransformers

Project Overview

GenoBench is an open-source benchmarking framework published on PyPI (v1.0.0) designed to systematically evaluate and profile Genomic Foundation Models (GFMs).

The tool currently supports models like HyenaDNA-tiny and DNABERT-2, enabling researchers to measure both genomic task accuracy and hardware-level performance metrics (such as memory footprint, token throughput, and latency).


Inspiration & Research Origins

GenoBench grew out of a master’s thesis research direction exploring hardware-aware inference of linear-time genomic language models.

In genomics, sequence lengths can scale to hundreds of thousands of base pairs. Standard Transformer models scale quadratically ($O(N^2)$) in time and memory complexity relative to sequence length, making them extremely expensive.

This research profiles and compares alternate architectures:

  1. State Space Models (SSMs) & Mamba: Linear-time architectures designed to capture long-range dependencies efficiently.
  2. HyenaDNA: Sub-quadratic convolutions designed specifically for long genomic contexts.
  3. Optimized Transformers (e.g., DNABERT-2): Utilizing flash attention and specialized tokenization to mitigate quadratic overhead.

Key Features

  • Unified Model Loader: Load different GFM architectures (HyenaDNA, DNABERT, etc.) with a single line of code.
  • Accuracy Profiling: Out-of-the-box support for genomics tasks (e.g., promoter detection, splice site classification, enhancer identification).
  • Hardware-Aware Metrics: Real-time logging of GPU/CPU memory usage, latency, and tokens-per-second during both inference and fine-tuning.
  • Extensible CLI: Run benchmarks from the command line and export reports in CSV, JSON, or markdown formats.
# Example command to benchmark DNABERT-2 on CUDA
genobench run --model dnabert-2 --task promoter-detection --device cuda --export report.json

How to Get Started

You can install GenoBench directly from PyPI:

pip install genobench

And run a quick evaluation test:

import genobench

# Benchmark HyenaDNA-tiny latency on CUDA
metrics = genobench.benchmark(
    model_name="hyenadna-tiny",
    task="enhancer-classification",
    device="cuda"
)

print(f"Throughput: {metrics.tokens_per_sec:.2f} tokens/second")
print(f"Peak Memory: {metrics.peak_memory_mb:.2f} MB")