Ad

Is There A Memory-efficient Way Of Inserting Zero-columns And Rows Into A Numpy Array?

- 1 answer

I have a big (symmetric) numpy matrix arr. It is of shape arr.shape = (50_000, 50_000). I want to insert some zero-rows/columns (in a symmetric way). Let's say the amount of rows / columns I want to insert is maybe 123.

Small Example

import numpy as np

# Create symmetric square matrix
size = 3
arr = np.array(list(range(size**2))).reshape(size, size)
arr = arr + arr.T

# insert zero columns
# Each "1" represents a column from the original matrix, e.g.
# the first 1 is the first column of arr, the second 1 the second column of arr
# and so on
insert_cols = [1, 0, 0, 1, 0, 1, 0, 0]

# insert the zero rows / columns
current_index = 0
for col in insert_cols:
    if col == 0:
        arr = np.insert(arr, current_index, 0, axis=0)
        arr = np.insert(arr, current_index, 0, axis=1)
    current_index += 1

print(arr)

If I understand np.insert correctly, then this code creates a copy of the array and copies the content all the time.

Question

I thought maybe this might be simpler / more efficient with one of the sparse matrix classes? Is there another way to make this more efficient?

Ad

Answer

Given insert_cols, we can do something like this -

n = len(insert_cols)
out = np.zeros((n,n),arr.dtype)
idx = np.flatnonzero(insert_cols)
out[np.ix_(idx,idx)] = arr # or out[idx[:,None],idx] = arr

Alternatively, use a boolean version for indexing. Hence -

insert_cols_bool = np.asarray(insert_cols, dtype=bool)

Then, use insert_cols_bool in place of idx.


Sparse-matrix

To be more memory-efficient, we can store the output as a sparse matrix -

from scipy.sparse import coo_matrix

l = len(idx)
r,c = np.broadcast_to(idx[:,None],(l,l)).ravel(),np.broadcast_to(idx,(l,l)).ravel()
out = coo_matrix((arr.ravel(), (r,c)), shape=(n,n))
Ad
source: stackoverflow.com
Ad