katzlabbrandeis

Catching the Drift: Enhancing Drift Detection Using Spike Counts

Date: January 21, 2025
Contributors: Abuzar Mahmood, abuzarmahmood
PR: https://github.com/katzlabbrandeis/blech_clust/pull/315

Introduction

In the field of neural data analysis, detecting drifts is a crucial task. A recent pull request on the ‘blech_clust’ repository by contributors Abuzar Mahmood and abuzarmahmood showcases an improved method for this task. Instead of performing drift detection on the PCA of firing rates, the updated code now performs the task on spike-time histograms. The changes not only enhance drift analysis, but also improve the ELBO drift plotting and calculations.

Analyzing the Changes

The key file that was changed in this pull request is utils/qa_utils/elbo_drift.py. This file is part of the ‘qa_utils’ module, a collection of utility functions used for quality assurance purposes in the ‘blech_clust’ package.

One of the principal changes in the code is the switch from PCA of firing rates to spike-time histograms for drift detection. This change allows the code to better account for variance in the data, thereby improving the accuracy of drift detection.

Another major change is the addition of flexible changepoints and CSV export functionality. This change significantly enhances the capability of the drift analysis process by allowing it to adapt to a wider range of data sets and facilitating easier data export and manipulation.

The code also includes an improved ELBO drift plotting and calculations. This makes the output of the code easier to interpret and use in downstream analyses.

-from tqdm import tqdm, trange
-import sys
-import os
-from pymc.variational.callbacks import CheckParametersConvergence
-import pymc as pm
-import pytensor.tensor as tt
-import numpy as np
-from sklearn.decomposition import PCA
-import matplotlib.pyplot as plt
-import pandas as pd
-import time
+from scipy.stats import zscore
+import tables
 import seaborn as sns
+import time
+import pandas as pd
+import matplotlib.pyplot as plt
+from sklearn.decomposition import PCA
+import numpy as np
+import pytensor.tensor as tt
+import pymc as pm
+from pymc.variational.callbacks import CheckParametersConvergence
+import os
+import sys
+from tqdm import tqdm, trange
+import argparse
+parser = argparse.ArgumentParser()
+parser.add_argument('dir_name', type=str, help='Directory containing data')
+parser.add_argument('--force', action='store_true', help='Force re-fitting')
+args = parser.parse_args()

Impact and Benefits

The updates to the drift detection method not only improve the accuracy of the process but also enhance its versatility. By using spike-time histograms, this code can handle more complex data sets and provide more accurate results.

The addition of flexible changepoints and CSV export functionality increases the code’s adaptability and usability. Researchers can now apply the code to a wider array of data sets, and the CSV export feature allows for easy data sharing and manipulation.

The improved ELBO drift plotting and calculations provide clearer visual representation of the data, which in turn aids interpretation and downstream analysis.

Conclusion

This pull request introduces vital updates to the ‘blech_clust’ repository, enhancing its ability to detect drifts in neural data. The changes not only improve the accuracy of the process but also increase the code’s versatility and user-friendliness. By constantly refining and improving our tools, we can continue to advance in our understanding of neural data.