Problem:
I can't figure out how to do a Two-sample KS test in Scipy.
After reading the documentation scipy kstest
I can see how to test where a distribution is identical to standard normal distribution
from scipy.stats import kstest
import numpy as np
x = np.random.normal(0,1,1000)
test_stat = kstest(x, 'norm')
#>>> test_stat
#(0.021080234718821145, 0.76584491300591395)
Which means that at p-value of 0.76 we can not reject the null hypothesis that the two distributions are identical.
However, I want to compare two distributions and see if I can reject the null hypothesis that they are identical, something like:
from scipy.stats import kstest
import numpy as np
x = np.random.normal(0,1,1000)
z = np.random.normal(1.1,0.9, 1000)
and test whether x and z are identical
I tried the naive:
test_stat = kstest(x, z)
and got the following error:
TypeError: 'numpy.ndarray' object is not callable
Is there a way to do a two-sample KS test in Python? If so, how should I do it?
Thank You in Advance
A:

from scipy import stats
import numpy as np
np.random.seed(42)
x = np.random.normal(0, 1, 1000)
y = np.random.normal(0, 1, 1000)

BEGIN SOLUTION

[insert]

END SOLUTION

print(statistic, p_value)


Problem:
I have the following data frame:
import pandas as pd
import io
from scipy import stats
temp=u"""probegenes,sample1,sample2,sample3
1415777_at Pnliprp1,20,0.00,11
1415805_at Clps,17,0.00,55
1415884_at Cela3b,47,0.00,100"""
df = pd.read_csv(io.StringIO(temp),index_col='probegenes')
df
It looks like this
                     sample1  sample2  sample3
probegenes
1415777_at Pnliprp1       20        0       11
1415805_at Clps           17        0       55
1415884_at Cela3b         47        0      100
What I want to do is too perform row-zscore calculation using SCIPY. AND I want to show data and zscore together in a single dataframe. At the end of the day. the result will look like:
                               sample1  sample2  sample3
probegenes
1415777_at Pnliprp1   data     20		  0			11
					zscore	  1.18195176 -1.26346568  0.08151391
1415805_at Clps		  data     17		  0			55
					zscore   -0.30444376 -1.04380717  1.34825093
1415884_at Cela3b	  data     47		  0			100
					zscore   -0.04896043 -1.19953047  1.2484909
A:

import pandas as pd
import io
from scipy import stats

temp=u"""probegenes,sample1,sample2,sample3
1415777_at Pnliprp1,20,0.00,11
1415805_at Clps,17,0.00,55
1415884_at Cela3b,47,0.00,100"""
df = pd.read_csv(io.StringIO(temp),index_col='probegenes')

BEGIN SOLUTION

[insert]

END SOLUTION

print(result)


Problem:
I have two data points on a 2-D image grid and the value of some quantity of interest at these two points is known.
For example:
Let us consider the point being x=(2,2). Then considering a 4-grid neighborhood we have points x_1=(1,2), x_2=(2,3), x_3=(3,2), x_4=(2,1) as neighbours of x. Suppose the value of some quantity of interest at these points be y=5, y_1=7, y_2=8, y_3= 10, y_4 = 3. Through interpolation, I want to find y at a sub-pixel value, say at (2.7, 2.3). The above problem can be represented with numpy arrays as follows.
x = [(2,2), (1,2), (2,3), (3,2), (2,1)]
y = [5,7,8,10,3]
How to use numpy/scipy linear interpolation to do this? I want result from griddata in scipy.
A:

import scipy.interpolate
x = [(2,2), (1,2), (2,3), (3,2), (2,1)]
y = [5,7,8,10,3]
eval = [(2.7, 2.3)]

BEGIN SOLUTION

[insert]

END SOLUTION

print(result)


Problem:
I have a data-set which contains many numerical and categorical values, and I want to only test for outlying values on the numerical columns and remove rows based on those columns.
I am trying it like this:
df = df[(np.abs(stats.zscore(df)) < 3).all(axis=1)]
Where it will remove all outlying values in all columns, however of course because I have categorical columns I am met with the following error:
TypeError: unsupported operand type(s) for +: 'float' and 'str'
I know the solution above works because if I limit my df to only contain numeric columns it all works fine but I don't want to lose the rest of the information in my dataframe in the process of evaluating outliers from numeric columns.
A:

from scipy import stats
import pandas as pd
import numpy as np
LETTERS = list('ABCDEFGHIJKLMNOPQRSTUVWXYZ')
df = pd.DataFrame({'NUM1': np.random.randn(50)*100,
                   'NUM2': np.random.uniform(0,1,50),
                   'NUM3': np.random.randint(100, size=50),
                   'CAT1': ["".join(np.random.choice(LETTERS,1)) for _ in range(50)],
                   'CAT2': ["".join(np.random.choice(['pandas', 'r', 'julia', 'sas', 'stata', 'spss'],1)) for _ in range(50)],
                   'CAT3': ["".join(np.random.choice(['postgres', 'mysql', 'sqlite', 'oracle', 'sql server', 'db2'],1)) for _ in range(50)]
                  })

BEGIN SOLUTION

[insert]

END SOLUTION

print(df)


Problem:

I'm trying to integrate X (X ~ N(u, o2)) to calculate the probability up to position `x`.
However I'm running into an error of:
Traceback (most recent call last):
  File "<ipython console>", line 1, in <module>
  File "siestats.py", line 349, in NormalDistro
    P_inner = scipy.integrate(NDfx,-dev,dev)
TypeError: 'module' object is not callable
My code runs this:
# Definition of the mathematical function:
def NDfx(x):
    return((1/math.sqrt((2*math.pi)))*(math.e**((-.5)*(x**2))))
# This Function normailizes x, u, and o2 (position of interest, mean and st dev)
# and then calculates the probability up to position 'x'
def NormalDistro(u,o2,x):
    dev = abs((x-u)/o2)
    P_inner = scipy.integrate(NDfx,-dev,dev)
    P_outer = 1 - P_inner
    P = P_inner + P_outer/2
    return(P)

A:

import scipy.integrate
import math
import numpy as np
def NDfx(x):
    return((1/math.sqrt((2*math.pi)))*(math.e**((-.5)*(x**2))))
def f(x = 2.5, u = 1, o2 = 3):

BEGIN SOLUTION

[insert]

END SOLUTION

    return prob


Problem:
I have a raster with a set of unique ID patches/regions which I've converted into a two-dimensional Python numpy array. I would like to calculate pairwise Euclidean distances between all regions to obtain the minimum distance separating the nearest edges of each raster patch. As the array was originally a raster, a solution needs to account for diagonal distances across cells (I can always convert any distances measured in cells back to metres by multiplying by the raster resolution).
I've experimented with the cdist function from scipy.spatial.distance as suggested in this answer to a related question, but so far I've been unable to solve my problem using the available documentation. As an end result I would ideally have a N*N array in the form of "from ID, to ID, distance", including distances between all possible combinations of regions.
Here's a sample dataset resembling my input data:
import numpy as np
import matplotlib.pyplot as plt
# Sample study area array
example_array = np.array([[0, 0, 0, 2, 2, 0, 0, 0, 0, 0, 0, 0],
                          [0, 0, 2, 0, 2, 2, 0, 6, 0, 3, 3, 3],
                          [0, 0, 0, 0, 2, 2, 0, 0, 0, 3, 3, 3],
                          [0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 3, 0],
                          [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 3],
                          [1, 1, 0, 0, 0, 0, 0, 0, 3, 3, 3, 3],
                          [1, 1, 1, 0, 0, 0, 3, 3, 3, 0, 0, 3],
                          [1, 1, 1, 0, 0, 0, 3, 3, 3, 0, 0, 0],
                          [1, 1, 1, 0, 0, 0, 3, 3, 3, 0, 0, 0],
                          [1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                          [1, 0, 1, 0, 0, 0, 0, 5, 5, 0, 0, 0],
                          [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4]])
# Plot array
plt.imshow(example_array, cmap="spectral", interpolation='nearest')
A:

import numpy as np
import scipy.spatial.distance
example_array = np.array([[0, 0, 0, 2, 2, 0, 0, 0, 0, 0, 0, 0],
                          [0, 0, 2, 0, 2, 2, 0, 6, 0, 3, 3, 3],
                          [0, 0, 0, 0, 2, 2, 0, 0, 0, 3, 3, 3],
                          [0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 3, 0],
                          [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 3],
                          [1, 1, 0, 0, 0, 0, 0, 0, 3, 3, 3, 3],
                          [1, 1, 1, 0, 0, 0, 3, 3, 3, 0, 0, 3],
                          [1, 1, 1, 0, 0, 0, 3, 3, 3, 0, 0, 0],
                          [1, 1, 1, 0, 0, 0, 3, 3, 3, 0, 0, 0],
                          [1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                          [1, 0, 1, 0, 0, 0, 0, 5, 5, 0, 0, 0],
                          [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4]])

BEGIN SOLUTION

[insert]

END SOLUTION

print(result)


Problem:
I am able to interpolate the data points (dotted lines), and am looking to extrapolate them in both direction.
How can I extrapolate these curves in Python with NumPy/SciPy?
The code I used for the interpolation is given below,
import numpy as np
import matplotlib.pyplot as plt
from scipy import interpolate
x = np.array([[0.12, 0.11, 0.1, 0.09, 0.08],
              [0.13, 0.12, 0.11, 0.1, 0.09],
              [0.15, 0.14, 0.12, 0.11, 0.1],
              [0.17, 0.15, 0.14, 0.12, 0.11],
              [0.19, 0.17, 0.16, 0.14, 0.12],
              [0.22, 0.19, 0.17, 0.15, 0.13],
              [0.24, 0.22, 0.19, 0.16, 0.14],
              [0.27, 0.24, 0.21, 0.18, 0.15],
              [0.29, 0.26, 0.22, 0.19, 0.16]])
y = np.array([[71.64, 78.52, 84.91, 89.35, 97.58],
              [66.28, 73.67, 79.87, 85.36, 93.24],
              [61.48, 69.31, 75.36, 81.87, 89.35],
              [57.61, 65.75, 71.7, 79.1, 86.13],
              [55.12, 63.34, 69.32, 77.29, 83.88],
              [54.58, 62.54, 68.7, 76.72, 82.92],
              [56.58, 63.87, 70.3, 77.69, 83.53],
              [61.67, 67.79, 74.41, 80.43, 85.86],
              [70.08, 74.62, 80.93, 85.06, 89.84]])
plt.figure(figsize = (5.15,5.15))
plt.subplot(111)
for i in range(5):
    x_val = np.linspace(x[0, i], x[-1, i], 100)
    x_int = np.interp(x_val, x[:, i], y[:, i])
    tck = interpolate.splrep(x[:, i], y[:, i], k = 2, s = 4)
    y_int = interpolate.splev(x_val, tck, der = 0)
    plt.plot(x[:, i], y[:, i], linestyle = '', marker = 'o')
    plt.plot(x_val, y_int, linestyle = ':', linewidth = 0.25, color =  'black')
plt.xlabel('X')
plt.ylabel('Y')
plt.show()

That seems only work for interpolation.
I want to use B-spline (with the same parameters setting as in the code) in scipy to do extrapolation. The result should be (5, 100) array containing f(x_val) for each group of x, y(just as shown in the code).

A:

from scipy import interpolate
import numpy as np
x = np.array([[0.12, 0.11, 0.1, 0.09, 0.08],
              [0.13, 0.12, 0.11, 0.1, 0.09],
              [0.15, 0.14, 0.12, 0.11, 0.1],
              [0.17, 0.15, 0.14, 0.12, 0.11],
              [0.19, 0.17, 0.16, 0.14, 0.12],
              [0.22, 0.19, 0.17, 0.15, 0.13],
              [0.24, 0.22, 0.19, 0.16, 0.14],
              [0.27, 0.24, 0.21, 0.18, 0.15],
              [0.29, 0.26, 0.22, 0.19, 0.16]])
y = np.array([[71.64, 78.52, 84.91, 89.35, 97.58],
              [66.28, 73.67, 79.87, 85.36, 93.24],
              [61.48, 69.31, 75.36, 81.87, 89.35],
              [57.61, 65.75, 71.7, 79.1, 86.13],
              [55.12, 63.34, 69.32, 77.29, 83.88],
              [54.58, 62.54, 68.7, 76.72, 82.92],
              [56.58, 63.87, 70.3, 77.69, 83.53],
              [61.67, 67.79, 74.41, 80.43, 85.86],
              [70.08, 74.62, 80.93, 85.06, 89.84]])
x_val = np.linspace(-1, 1, 100)

BEGIN SOLUTION

[insert]

END SOLUTION

print(result)


Problem:
I would like to write a program that solves the definite integral below in a loop which considers a different value of the constant c per iteration.
I would then like each solution to the integral to be outputted into a new array.
How do I best write this program in python?
∫2cxdx with limits between 0 and 1.
from scipy import integrate
integrate.quad
Is acceptable here. My major struggle is structuring the program.
Here is an old attempt (that failed)
# import c
fn = 'cooltemp.dat'
c = loadtxt(fn,unpack=True,usecols=[1])
I=[]
for n in range(len(c)):
    # equation
    eqn = 2*x*c[n]
    # integrate
    result,error = integrate.quad(lambda x: eqn,0,1)
    I.append(result)
I = array(I)
A:

import scipy.integrate
c = 5
low = 0
high = 1

BEGIN SOLUTION

[insert]

END SOLUTION

print(result)


Problem:
I have a sparse 988x1 vector (stored in col, a column in a csr_matrix) created through scipy.sparse. Is there a way to gets its mean and standard deviation without having to convert the sparse matrix to a dense one?
numpy.mean seems to only work for dense vectors.

A:

import numpy as np
from scipy.sparse import csr_matrix

np.random.seed(10)
arr = np.random.randint(4,size=(988,988))
sA = csr_matrix(arr)
col = sA.getcol(0)

BEGIN SOLUTION

[insert]

END SOLUTION

print(mean)
print(standard_deviation)


Problem:
I’m trying to solve a simple ODE to visualise the temporal response, which works well for constant input conditions using the new solve_ivp integration API in SciPy. For example:
def dN1_dt_simple(t, N1):
    return -100 * N1
sol = solve_ivp(fun=dN1_dt_simple, t_span=[0, 100e-3], y0=[N0,])
However, I wonder is it possible to plot the response to a time-varying input? For instance, rather than having y0 fixed at N0, can I find the response to a simple sinusoid? Specifically, I want to add `t-sin(t) if 0 < t < 2pi else 2pi` to original y. The result I want is values of solution at time points.
Is there a compatible way to pass time-varying input conditions into the API?
A:

import scipy.integrate
import numpy as np
N0 = 1
time_span = [0, 10]

BEGIN SOLUTION

[insert]

END SOLUTION

result = sol.y
print(result)


Problem:
How does one convert a left-tailed p-value to a z_score from the Z-distribution (standard normal distribution, Gaussian distribution)? I have yet to find the magical function in Scipy's stats module to do this, but one must be there.
A:

import numpy as np
import scipy.stats
p_values = [0.1, 0.225, 0.5, 0.75, 0.925, 0.95]

BEGIN SOLUTION

[insert]

END SOLUTION

print(z_scores)


Problem:

Using scipy, is there an easy way to emulate the behaviour of MATLAB's dctmtx function which returns a NxN (ortho-mode normed) DCT matrix for some given N? There's scipy.fftpack.dctn but that only applies the DCT. Do I have to implement this from scratch if I don't want use another dependency besides scipy?
A:

import numpy as np
import scipy.fft as sf
N = 8

BEGIN SOLUTION

[insert]

END SOLUTION

print(result)


Problem:
I have a sparse matrix in csr format (which makes sense for my purposes, as it has lots of rows but relatively few columns, ~8million x 90).
My question is, what's the most efficient way to access particular values from the matrix given lists of row,column indices? I can quickly get a row using matrix.getrow(row), but this also returns 1-row sparse matrix, and accessing the value at a particular column seems clunky. The only reliable method I've found to get a particular matrix value, given the row and column, is:
getting the row vector, converting to dense array, and fetching the element on column.

But this seems overly verbose and complicated. and I don't want to change it to dense matrix to keep the efficiency.
for example, I want to fetch elements at (2, 3) and (1, 0), so row = [2, 1], and column = [3, 0].
The result should be a list or 1-d array like: [matirx[2, 3], matrix[1, 0]]
Is there a simpler/faster method I'm missing?

A:

import numpy as np
from scipy.sparse import csr_matrix

arr = np.array([[1,2,3,4],[5,6,7,8],[9,10,11,12],[13,14,15,16]])
M = csr_matrix(arr)
row = [2, 1]
column = [3, 0]

BEGIN SOLUTION

[insert]

END SOLUTION

print(result)


Problem:
I am working with a 2D numpy array made of 512x512=262144 values. Such values are of float type and range from 0.0 to 1.0. The array has an X,Y coordinate system which originates in the top left corner: thus, position (0,0) is in the top left corner, while position (512,512) is in the bottom right corner.
This is how the 2D array looks like (just an excerpt):
X,Y,Value
0,0,0.482
0,1,0.49
0,2,0.496
0,3,0.495
0,4,0.49
0,5,0.489
0,6,0.5
0,7,0.504
0,8,0.494
0,9,0.485

I would like to be able to:
Count the number of regions of cells which value exceeds a given threshold, i.e. 0.75;

Note: If two elements touch horizontally, vertically or diagnoally, they belong to one region.

A:

import numpy as np
from scipy import ndimage

np.random.seed(10)
gen = np.random.RandomState(0)
img = gen.poisson(2, size=(512, 512))
img = ndimage.gaussian_filter(img.astype(np.double), (30, 30))
img -= img.min()
img /= img.max()
threshold = 0.75

BEGIN SOLUTION

[insert]

END SOLUTION

print(result)


Problem:
I have a sparse matrix in csr format (which makes sense for my purposes, as it has lots of rows but relatively few columns, ~8million x 90).
My question is, what's the most efficient way to access a particular value from the matrix given a row,column tuple? I can quickly get a row using matrix.getrow(row), but this also returns 1-row sparse matrix, and accessing the value at a particular column seems clunky.
The only reliable method I've found to get a particular matrix value, given the row and column, is:
getting the row vector, converting to dense array, and fetching the element on column.

But this seems overly verbose and complicated. and I don't want to change it to dense matrix to keep the efficiency.
Is there a simpler/faster method I'm missing?

A:

import numpy as np
from scipy.sparse import csr_matrix

arr = np.array([[1,2,3,4],[5,6,7,8],[9,10,11,12],[13,14,15,16]])
M = csr_matrix(arr)
row = 2
column = 3

BEGIN SOLUTION

[insert]

END SOLUTION

print(result)


Problem:
Basically, I am just trying to do a simple matrix multiplication, specifically, extract each column of it and normalize it by dividing it with its length.
    #csc sparse matrix
    self.__WeightMatrix__ = self.__WeightMatrix__.tocsc()
    #iterate through columns
    for Col in xrange(self.__WeightMatrix__.shape[1]):
       Column = self.__WeightMatrix__[:,Col].data
       List = [x**2 for x in Column]
       #get the column length
       Len = math.sqrt(sum(List))
       #here I assumed dot(number,Column) would do a basic scalar product
       dot((1/Len),Column)
       #now what? how do I update the original column of the matrix, everything that have been returned are copies, which drove me nuts and missed pointers so much
I've searched through the scipy sparse matrix documentations and got no useful information. I was hoping for a function to return a pointer/reference to the matrix so that I can directly modify its value. Thanks
A:

from scipy import sparse
import numpy as np
import math
sa = sparse.random(10, 10, density = 0.3, format = 'csc', random_state = 42)

BEGIN SOLUTION

[insert]

END SOLUTION

print(sa)


Problem:
I am looking for a way to convert a nXaXb numpy array into a block diagonal matrix. I have already came across scipy.linalg.block_diag, the down side of which (for my case) is it requires each blocks of the matrix to be given separately. However, this is challenging when n is very high, so to make things more clear lets say I have a
import numpy as np
a = np.random.rand(3,2,2)
array([[[ 0.33599705,  0.92803544],
        [ 0.6087729 ,  0.8557143 ]],
       [[ 0.81496749,  0.15694689],
        [ 0.87476697,  0.67761456]],
       [[ 0.11375185,  0.32927167],
        [ 0.3456032 ,  0.48672131]]])

what I want to achieve is something the same as
from scipy.linalg import block_diag
block_diag(a[0], a[1],a[2])
array([[ 0.33599705,  0.92803544,  0.        ,  0.        ,  0.        ,   0.        ],
       [ 0.6087729 ,  0.8557143 ,  0.        ,  0.        ,  0.        ,   0.        ],
       [ 0.        ,  0.        ,  0.81496749,  0.15694689,  0.        ,   0.        ],
       [ 0.        ,  0.        ,  0.87476697,  0.67761456,  0.        ,   0.        ],
       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.11375185,   0.32927167],
       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.3456032 ,   0.48672131]])

This is just as an example in actual case a has hundreds of elements.

A:

import numpy as np
from scipy.linalg import block_diag
np.random.seed(10)
a = np.random.rand(100,2,2)

BEGIN SOLUTION

[insert]

END SOLUTION

print(result)


Problem:
First off, I'm no mathmatician. I admit that. Yet I still need to understand how ScyPy's sparse matrices work arithmetically in order to switch from a dense NumPy matrix to a SciPy sparse matrix in an application I have to work on. The issue is memory usage. A large dense matrix will consume tons of memory.
The formula portion at issue is where a matrix is added to a scalar.
A = V + x
Where V is a square sparse matrix (its large, say 60,000 x 60,000). x is a float.
What I want is that x will only be added to non-zero values in V.
With a SciPy, not all sparse matrices support the same features, like scalar addition. dok_matrix (Dictionary of Keys) supports scalar addition, but it looks like (in practice) that it's allocating each matrix entry, effectively rendering my sparse dok_matrix as a dense matrix with more overhead. (not good)
The other matrix types (CSR, CSC, LIL) don't support scalar addition.
I could try constructing a full matrix with the scalar value x, then adding that to V. I would have no problems with matrix types as they all seem to support matrix addition. However I would have to eat up a lot of memory to construct x as a matrix, and the result of the addition could end up being fully populated matrix as well.
There must be an alternative way to do this that doesn't require allocating 100% of a sparse matrix. I’d like to solve the problem on coo matrix first.
I'm will to accept that large amounts of memory are needed, but I thought I would seek some advice first. Thanks.
A:
<code>
from scipy import sparse
V = sparse.random(10, 10, density = 0.05, format = 'coo', random_state = 42)
x = 100
</code>
V = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
What is the canonical way to check if a SciPy CSR matrix is empty (i.e. contains only zeroes)?
I use nonzero():
def is_csr_matrix_only_zeroes(my_csr_matrix):
    return(len(my_csr_matrix.nonzero()[0]) == 0)
from scipy.sparse import csr_matrix
print(is_csr_matrix_only_zeroes(csr_matrix([[1,2,0],[0,0,3],[4,0,5]])))
print(is_csr_matrix_only_zeroes(csr_matrix([[0,0,0],[0,0,0],[0,0,0]])))
print(is_csr_matrix_only_zeroes(csr_matrix((2,3))))
print(is_csr_matrix_only_zeroes(csr_matrix([[0,0,0],[0,1,0],[0,0,0]])))
outputs
False
True
True
False
but I wonder whether there exist more direct or efficient ways, i.e. just get True or False?
A:
<code>
from scipy import sparse
sa = sparse.random(10, 10, density = 0.01, format = 'csr')
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I have been trying to get the result of a lognormal distribution using Scipy. I already have the Mu and Sigma, so I don't need to do any other prep work. If I need to be more specific (and I am trying to be with my limited knowledge of stats), I would say that I am looking for the cumulative function (cdf under Scipy). The problem is that I can't figure out how to do this with just the mean and standard deviation on a scale of 0-1 (ie the answer returned should be something from 0-1). I'm also not sure which method from dist, I should be using to get the answer. I've tried reading the documentation and looking through SO, but the relevant questions (like this and this) didn't seem to provide the answers I was looking for.
Here is a code sample of what I am working with. Thanks. Here mu and stddev stands for mu and sigma in probability density function of lognorm.
from scipy.stats import lognorm
stddev = 0.859455801705594
mu = 0.418749176686875
total = 37
dist = lognorm.cdf(total,mu,stddev)
UPDATE:
So after a bit of work and a little research, I got a little further. But I still am getting the wrong answer. The new code is below. According to R and Excel, the result should be .7434, but that's clearly not what is happening. Is there a logic flaw I am missing?
stddev = 2.0785
mu = 1.744
x = 25
dist = lognorm([mu],loc=stddev)
dist.cdf(x)  # yields=0.96374596, expected=0.7434
A:
<code>
import numpy as np
from scipy import stats
stddev = 2.0785
mu = 1.744
x = 25
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I would like to resample a numpy array as suggested here Resampling a numpy array representing an image however this resampling will do so by a factor i.e.
x = np.arange(9).reshape(3,3)
print scipy.ndimage.zoom(x, 2, order=1)
Will create a shape of (6,6) but how can I resample an array to its best approximation within a (4,6),(6,8) or (6,10) shape for instance?
A:
<code>
import numpy as np
import scipy.ndimage
x = np.arange(9).reshape(3, 3)
shape = (6, 8)
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I simulate times in the range 0 to T according to a Poisson process. The inter-event times are exponential and we know that the distribution of the times should be uniform in the range 0 to T.
def poisson_simul(rate, T):
    time = random.expovariate(rate)
    times = [0]
    while (times[-1] < T):
        times.append(time+times[-1])
        time = random.expovariate(rate)
    return times[1:]
I would simply like to run one of the tests for uniformity, for example the Kolmogorov-Smirnov test. I can't work out how to do this in scipy however. If I do
import random
from scipy.stats import kstest
times = poisson_simul(1, 100)
print kstest(times, "uniform")
it is not right . It gives me
(1.0, 0.0)
I just want to test the hypothesis that the points are uniformly chosen from the range 0 to T. How do you do this in scipy? The result should be KStest result.
A:
<code>
from scipy import stats
import random
import numpy as np
def poisson_simul(rate, T):
    time = random.expovariate(rate)
    times = [0]
    while (times[-1] < T):
        times.append(time+times[-1])
        time = random.expovariate(rate)
    return times[1:]
rate = 1.0
T = 100.0
times = poisson_simul(rate, T)
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I have this example of matrix by matrix multiplication using numpy arrays:
import numpy as np
m = np.array([[1,2,3],[4,5,6],[7,8,9]])
c = np.array([0,1,2])
m * c
array([[ 0,  2,  6],
       [ 0,  5, 12],
       [ 0,  8, 18]])
How can i do the same thing if m is scipy sparse CSR matrix? The result should be csr_matrix as well.
This gives dimension mismatch:
sp.sparse.csr_matrix(m)*sp.sparse.csr_matrix(c)

A:
<code>
from scipy import sparse
import numpy as np
sa = sparse.csr_matrix(np.array([[1,2,3],[4,5,6],[7,8,9]]))
sb = sparse.csr_matrix(np.array([0,1,2]))
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:

I am having a problem with minimization procedure. Actually, I could not create a correct objective function for my problem.
Problem definition
•	My function: yn = a_11*x1**2 + a_12*x2**2 + ... + a_m*xn**2,where xn- unknowns, a_m - coefficients. n = 1..N, m = 1..M
•	In my case, N=5 for x1,..,x5 and M=3 for y1, y2, y3.
I need to find the optimum: x1, x2,...,x5 so that it can satisfy the y
My question:
•	How to solve the question using scipy.optimize?
My code:   (tried in lmfit, but return errors. Therefore I would ask for scipy solution)
import numpy as np
from lmfit import Parameters, minimize
def func(x,a):
    return np.dot(a, x**2)
def residual(pars, a, y):
    vals = pars.valuesdict()
    x = vals['x']
    model = func(x,a)
    return (y - model)**2
def main():
    # simple one: a(M,N) = a(3,5)
    a = np.array([ [ 0, 0, 1, 1, 1 ],
                   [ 1, 0, 1, 0, 1 ],
                   [ 0, 1, 0, 1, 0 ] ])
    # true values of x
    x_true = np.array([10, 13, 5, 8, 40])
    # data without noise
    y = func(x_true,a)
    #************************************
    # Apriori x0
    x0 = np.array([2, 3, 1, 4, 20])
    fit_params = Parameters()
    fit_params.add('x', value=x0)
    out = minimize(residual, fit_params, args=(a, y))
    print out
if __name__ == '__main__':
main()
Result should be optimal x array. The method I hope to use is L-BFGS-B, with added lower bounds on x.

A:

<code>
import scipy.optimize
import numpy as np
np.random.seed(42)
a = np.random.rand(3,5)
x_true = np.array([10, 13, 5, 8, 40])
y = a.dot(x_true ** 2)
x0 = np.array([2, 3, 1, 4, 20])
x_lower_bounds = x_true / 2
</code>
out = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I would like to write a program that solves the definite integral below in a loop which considers a different value of the constant c per iteration.
I would then like each solution to the integral to be outputted into a new array.
How do I best write this program in python?
∫2cxdx with limits between 0 and 1.
from scipy import integrate
integrate.quad
Is acceptable here. My major struggle is structuring the program.
Here is an old attempt (that failed)
# import c
fn = 'cooltemp.dat'
c = loadtxt(fn,unpack=True,usecols=[1])
I=[]
for n in range(len(c)):
    # equation
    eqn = 2*x*c[n]
    # integrate
    result,error = integrate.quad(lambda x: eqn,0,1)
    I.append(result)
I = array(I)
A:
<code>
import scipy.integrate
def f(c=5, low=0, high=1):
    # return the solution in this function
    # result = f(c=5, low=0, high=1)
    ### BEGIN SOLUTION


Problem:
I have the following data frame:
import pandas as pd
import io
from scipy import stats
temp=u"""probegenes,sample1,sample2,sample3
1415777_at Pnliprp1,20,0.00,11
1415805_at Clps,17,0.00,55
1415884_at Cela3b,47,0.00,100"""
df = pd.read_csv(io.StringIO(temp),index_col='probegenes')
df
It looks like this
                     sample1  sample2  sample3
probegenes
1415777_at Pnliprp1       20        0       11
1415805_at Clps           17        0       55
1415884_at Cela3b         47        0      100
What I want to do is too perform column-zscore calculation using SCIPY. AND I want to show data and zscore together in a single dataframe. For each element, I want to only keep 3 decimals places. At the end of the day. the result will look like:
                               sample1  sample2  sample3
probegenes
1415777_at Pnliprp1   data     20.000    0.000    11.000
					zscore	   -0.593    NaN    -1.220
1415805_at Clps		  data     17.000	0.000	55.000
					zscore     -0.815    NaN    -0.009
1415884_at Cela3b	  data     47.000	0.000	100.000
					zscore     1.408     NaN     1.229

A:
<code>
import pandas as pd
import io
import numpy as np
from scipy import stats

temp=u"""probegenes,sample1,sample2,sample3
1415777_at Pnliprp1,20,0.00,11
1415805_at Clps,17,0.00,55
1415884_at Cela3b,47,0.00,100"""
df = pd.read_csv(io.StringIO(temp),index_col='probegenes')
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
How to calculate kurtosis (the fourth standardized moment, according to Pearson’s definition) without bias correction?
I have tried scipy.stats.kurtosis, but it gives a different result. I followed the definition in mathworld.
A:
<code>
import numpy as np
a = np.array([   1. ,    2. ,    2.5,  400. ,    6. ,    0. ])
</code>
kurtosis_result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I have a set of data and I want to compare which line describes it best (polynomials of different orders, exponential or logarithmic).
I use Python and Numpy and for polynomial fitting there is a function polyfit().
How do I fit y = A + Blogx using polyfit()? The result should be an np.array of [A, B]
A:
<code>
import numpy as np
import scipy
x = np.array([1, 7, 20, 50, 79])
y = np.array([10, 19, 30, 35, 51])

</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I have an array of experimental values and a probability density function that supposedly describes their distribution:
def bekkers(x, a, m, d):
    p = a*np.exp((-1*(x**(1/3) - m)**2)/(2*d**2))*x**(-2/3)
    return(p)
I estimated the parameters of my function using scipy.optimize.curve_fit and now I need to somehow test the goodness of fit. I found a scipy.stats.kstest function which suposedly does exactly what I need, but it requires a continuous distribution function.
How do I get the result (statistic, pvalue) of KStest? I have some sample_data from fitted function, and parameters of it.
A:
<code>
import numpy as np
import scipy as sp
from scipy import integrate,stats
def bekkers(x, a, m, d):
    p = a*np.exp((-1*(x**(1/3) - m)**2)/(2*d**2))*x**(-2/3)
    return(p)
range_start = 1
range_end = 10
estimated_a, estimated_m, estimated_d = 1,1,1
sample_data = [1.5,1.6,1.8,2.1,2.2,3.3,4,6,8,9]
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I have the following data frame:
import pandas as pd
import io
from scipy import stats
temp=u"""probegenes,sample1,sample2,sample3
1415777_at Pnliprp1,20,0.00,11
1415805_at Clps,17,0.00,55
1415884_at Cela3b,47,0.00,100"""
df = pd.read_csv(io.StringIO(temp),index_col='probegenes')
df
It looks like this
                     sample1  sample2  sample3
probegenes
1415777_at Pnliprp1       20        0       11
1415805_at Clps           17        0       55
1415884_at Cela3b         47        0      100
What I want to do is too perform row-zscore calculation using SCIPY. At the end of the day. the result will look like:
                               sample1  sample2  sample3
probegenes
1415777_at Pnliprp1      1.18195176, -1.26346568,  0.08151391
1415805_at Clps         -0.30444376, -1.04380717,  1.34825093
1415884_at Cela3b        -0.04896043, -1.19953047,  1.2484909
A:
<code>
import pandas as pd
import io
from scipy import stats

temp=u"""probegenes,sample1,sample2,sample3
1415777_at Pnliprp1,20,0.00,11
1415805_at Clps,17,0.00,55
1415884_at Cela3b,47,0.00,100"""
df = pd.read_csv(io.StringIO(temp),index_col='probegenes')
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I have a raster with a set of unique ID patches/regions which I've converted into a two-dimensional Python numpy array. I would like to calculate pairwise Euclidean distances between all regions to obtain the minimum distance separating the nearest edges of each raster patch. As the array was originally a raster, a solution needs to account for diagonal distances across cells (I can always convert any distances measured in cells back to metres by multiplying by the raster resolution).
I've experimented with the cdist function from scipy.spatial.distance as suggested in this answer to a related question, but so far I've been unable to solve my problem using the available documentation. As an end result I would ideally have a N*N array in the form of "from ID, to ID, distance", including distances between all possible combinations of regions.
Here's a sample dataset resembling my input data:
import numpy as np
import matplotlib.pyplot as plt
# Sample study area array
example_array = np.array([[0, 0, 0, 2, 2, 0, 0, 0, 0, 0, 0, 0],
                          [0, 0, 2, 0, 2, 2, 0, 6, 0, 3, 3, 3],
                          [0, 0, 0, 0, 2, 2, 0, 0, 0, 3, 3, 3],
                          [0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 3, 0],
                          [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 3],
                          [1, 1, 0, 0, 0, 0, 0, 0, 3, 3, 3, 3],
                          [1, 1, 1, 0, 0, 0, 3, 3, 3, 0, 0, 3],
                          [1, 1, 1, 0, 0, 0, 3, 3, 3, 0, 0, 0],
                          [1, 1, 1, 0, 0, 0, 3, 3, 3, 0, 0, 0],
                          [1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                          [1, 0, 1, 0, 0, 0, 0, 5, 5, 0, 0, 0],
                          [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4]])
# Plot array
plt.imshow(example_array, cmap="spectral", interpolation='nearest')
A:
<code>
import numpy as np
import scipy.spatial.distance
example_arr = np.array([[0, 0, 0, 2, 2, 0, 0, 0, 0, 0, 0, 0],
                          [0, 0, 2, 0, 2, 2, 0, 6, 0, 3, 3, 3],
                          [0, 0, 0, 0, 2, 2, 0, 0, 0, 3, 3, 3],
                          [0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 3, 0],
                          [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 3],
                          [1, 1, 0, 0, 0, 0, 0, 0, 3, 3, 3, 3],
                          [1, 1, 1, 0, 0, 0, 3, 3, 3, 0, 0, 3],
                          [1, 1, 1, 0, 0, 0, 3, 3, 3, 0, 0, 0],
                          [1, 1, 1, 0, 0, 0, 3, 3, 3, 0, 0, 0],
                          [1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                          [1, 0, 1, 0, 0, 0, 0, 5, 5, 0, 0, 0],
                          [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4]])
def f(example_array = example_arr):
    # return the solution in this function
    # result = f(example_array)
    ### BEGIN SOLUTION


Problem:
According to the SciPy documentation it is possible to minimize functions with multiple variables, yet it doesn't tell how to optimize on such functions.
from scipy.optimize import minimize
from math import *
def f(c):
  return sqrt((sin(pi/2) + sin(0) + sin(c) - 2)**2 + (cos(pi/2) + cos(0) + cos(c) - 1)**2)
print minimize(f, 3.14/2 + 3.14/7)

The above code does try to minimize the function f, but for my task I need to minimize with respect to three variables, starting from `initial_guess`.
Simply introducing a second argument and adjusting minimize accordingly yields an error (TypeError: f() takes exactly 2 arguments (1 given)).
How does minimize work when minimizing with multiple variables.
I need to minimize f(a,b,c)=((a+b-c)-2)**2 + ((3*a-b-c))**2 + sin(b) + cos(b) + 4.
Result should be a list=[a,b,c], the parameters of minimized function.

A:
<code>
import scipy.optimize as optimize
from math import *

initial_guess = [-1, 0, -3]
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I just start learning Python. Here is a data frame:
a=pd.DataFrame({'A1':[0,1,2,3,2,1,6,0,1,1,7,10]})
Now I think this data follows multinomial distribution. So, 12 numbers means the frequency of 12 categories (category 0, 1, 2...). For example, the occurance of category 0 is 0. So, I hope to find all the parameters of multinomial given this data. In the end, we have the best parameters of multinomial (or we can say the best probility for every number). For example,
category:    0,      1,     2,     3,      4...
weights:    0.001,  0.1,   0.2,   0.12,   0.2...
So, I do not need a test data to predict. Could anyone give me some help?
I know that Maximum Likelihood Estimation is one of the most important procedure to get point estimation for parameters of a distribution. So how can I apply it to this question?
A:
<code>
import scipy.optimize as sciopt
import numpy as np
import pandas as pd
a=pd.DataFrame({'A1':[0,1,2,3,2,1,6,0,1,1,7,10]})
</code>
weights = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
Scipy offers many useful tools for root finding, notably fsolve. Typically a program has the following form:
def eqn(x, a, b):
    return x + 2*a - b**2
fsolve(eqn, x0=0.5, args = (a,b))
and will find a root for eqn(x) = 0 given some arguments a and b.
However, what if I have a problem where I want to solve for the a variable, giving the function arguments in x and b? Of course, I could recast the initial equation as
def eqn(a, x, b)
but this seems long winded and inefficient. Instead, is there a way I can simply set fsolve (or another root finding algorithm) to allow me to choose which variable I want to solve for?
Note that the result should be an array of roots for many (x, b) pairs.
A:
<code>
import numpy as np
from scipy.optimize import fsolve
def eqn(x, a, b):
    return x + 2*a - b**2

xdata = np.arange(4)+3
bdata = np.random.randint(0, 10, (4,))
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I simulate times in the range 0 to T according to a Poisson process. The inter-event times are exponential and we know that the distribution of the times should be uniform in the range 0 to T.
def poisson_simul(rate, T):
    time = random.expovariate(rate)
    times = [0]
    while (times[-1] < T):
        times.append(time+times[-1])
        time = random.expovariate(rate)
    return times[1:]
I would simply like to run one of the tests for uniformity, for example the Kolmogorov-Smirnov test. I can't work out how to do this in scipy however. If I do
import random
from scipy.stats import kstest
times = poisson_simul(1, 100)
print kstest(times, "uniform")
it is not right . It gives me
(1.0, 0.0)
I just want to test the hypothesis that the points are uniformly chosen from the range 0 to T. How do you do this in scipy? Another question is how to interpret the result? What I want is just `True` for unifomity or `False` vice versa. Suppose I want a confidence level of 95%.
A:
<code>
from scipy import stats
import random
import numpy as np
def poisson_simul(rate, T):
    time = random.expovariate(rate)
    times = [0]
    while (times[-1] < T):
        times.append(time+times[-1])
        time = random.expovariate(rate)
	return times[1:]
rate = 1.0
T = 100.0
times = poisson_simul(rate, T)
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
How can I extract the main diagonal(1-d array) of a sparse matrix? The matrix is created in scipy.sparse. I want equivalent of np.diagonal(), but for sparse matrix.

A:
<code>
import numpy as np
from scipy.sparse import csr_matrix

arr = np.random.rand(4, 4)
M = csr_matrix(arr)
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I simulate times in the range 0 to T according to a Poisson process. The inter-event times are exponential and we know that the distribution of the times should be uniform in the range 0 to T.
def poisson_simul(rate, T):
    time = random.expovariate(rate)
    times = [0]
    while (times[-1] < T):
        times.append(time+times[-1])
        time = random.expovariate(rate)
    return times[1:]
I would simply like to run one of the tests for uniformity, for example the Kolmogorov-Smirnov test. I can't work out how to do this in scipy however. If I do
import random
from scipy.stats import kstest
times = poisson_simul(1, 100)
print kstest(times, "uniform")
it is not right . It gives me
(1.0, 0.0)
I just want to test the hypothesis that the points are uniformly chosen from the range 0 to T. How do you do this in scipy? The result should be KStest result.
A:
<code>
from scipy import stats
import random
import numpy as np
def poisson_simul(rate, T):
    time = random.expovariate(rate)
    times = [0]
    while (times[-1] < T):
        times.append(time+times[-1])
        time = random.expovariate(rate)
    return times[1:]
example_rate = 1.0
example_T = 100.0
example_times = poisson_simul(example_rate, example_T)
def f(times = example_times, rate = example_rate, T = example_T):
    # return the solution in this function
    # result = f(times, rate, T)
    ### BEGIN SOLUTION


Problem:
I'm searching for examples of using scipy.optimize.line_search. I do not really understand how this function works with multivariable functions. I wrote a simple example
import scipy as sp
import scipy.optimize
def test_func(x):
    return (x[0])**2+(x[1])**2

def test_grad(x):
    return [2*x[0],2*x[1]]

sp.optimize.line_search(test_func,test_grad,[1.8,1.7],[-1.0,-1.0])
And I've got
File "D:\Anaconda2\lib\site-packages\scipy\optimize\linesearch.py", line 259, in phi
return f(xk + alpha * pk, *args)
TypeError: can't multiply sequence by non-int of type 'float'
The result should be the alpha value of line_search
A:
<code>
import scipy
import scipy.optimize
import numpy as np
def test_func(x):
    return (x[0])**2+(x[1])**2

def test_grad(x):
    return [2*x[0],2*x[1]]
starting_point = [1.8, 1.7]
direction = [-1, -1]
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
How does one convert a list of Z-scores from the Z-distribution (standard normal distribution, Gaussian distribution) to left-tailed p-values? Original data is sampled from X ~ N(mu, sigma). I have yet to find the magical function in Scipy's stats module to do this, but one must be there.
A:
<code>
import scipy.stats
import numpy as np
z_scores = [-3, -2, 0, 2, 2.5]
mu = 3
sigma = 4
</code>
p_values = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
Is there a simple and efficient way to make a sparse scipy matrix (e.g. lil_matrix, or csr_matrix) symmetric?
Currently I have a lil sparse matrix, and not both of sA[i,j] and sA[j,i] have element for any i,j.
When populating a large sparse co-occurrence matrix it would be highly inefficient to fill in [row, col] and [col, row] at the same time. What I'd like to be doing is:
for i in data:
    for j in data:
        if have_element(i, j):
            lil_sparse_matrix[i, j] = some_value
            # want to avoid this:
            # lil_sparse_matrix[j, i] = some_value
# this is what I'm looking for:
lil_sparse.make_symmetric()
and it let sA[i,j] = sA[j,i] for any i, j.

This is similar to stackoverflow's numpy-smart-symmetric-matrix question, but is particularly for scipy sparse matrices.

A:

import numpy as np
from scipy.sparse import lil_matrix
from scipy import sparse

M= sparse.random(10, 10, density=0.1, format='lil')

M = ... # put solution in this variable
BEGIN SOLUTION


Problem:
I have two csr_matrix, c1 and c2.

I want a new matrix
Feature = [c1
           c2].

That is, I want to concatenate c1 and c2 in vertical direction.

But I don't know how to represent the concatenation or how to form the format.

How can I achieve the matrix concatenation and still get the same type of matrix, i.e. a csr_matrix?

Any help would be appreciated.

A:
<code>
from scipy import sparse
c1 = sparse.csr_matrix([[0, 0, 1, 0], [2, 0, 0, 0], [0, 0, 0, 0]])
c2 = sparse.csr_matrix([[0, 3, 4, 0], [0, 0, 0, 5], [6, 7, 0, 8]])
</code>
Feature = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I'm trying to create a 2-dimensional array in Scipy/Numpy where each value represents the euclidean distance from the center.
I'm very new to Scipy, and would like to know if there's a more elegant, idiomatic way of doing the same thing. I found the scipy.spatial.distance.cdist function, which seems promising, but I'm at a loss regarding how to fit it into this problem.
def get_distance_2(y, x):
    mid = ...  # needs to be a array of the shape (rows, cols, 2)?
    return scipy.spatial.distance.cdist(scipy.dstack((y, x)), mid)
Just to clarify, what I'm looking for is something like this (for a 6 x 6 array). That is, to compute (Euclidean) distances from center point to every point in the image.
[[ 3.53553391  2.91547595  2.54950976  2.54950976  2.91547595  3.53553391]
 [ 2.91547595  2.12132034  1.58113883  1.58113883  2.12132034  2.91547595]
 [ 2.54950976  1.58113883  0.70710678  0.70710678  1.58113883  2.54950976]
 [ 2.54950976  1.58113883  0.70710678  0.70710678  1.58113883  2.54950976]
 [ 2.91547595  2.12132034  1.58113883  1.58113883  2.12132034  2.91547595]
 [ 3.53553391  2.91547595  2.54950976  2.54950976  2.91547595  3.53553391]]
A:
<code>
import numpy as np
from scipy.spatial import distance
shape = (6, 6)
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I have a raster with a set of unique ID patches/regions which I've converted into a two-dimensional Python numpy array. I would like to calculate pairwise Manhattan distances between all regions to obtain the minimum distance separating the nearest edges of each raster patch.
I've experimented with the cdist function from scipy.spatial.distance as suggested in this answer to a related question, but so far I've been unable to solve my problem using the available documentation. As an end result I would ideally have a N*N array in the form of "from ID, to ID, distance", including distances between all possible combinations of regions.
Here's a sample dataset resembling my input data:
import numpy as np
import matplotlib.pyplot as plt
# Sample study area array
example_array = np.array([[0, 0, 0, 2, 2, 0, 0, 0, 0, 0, 0, 0],
                          [0, 0, 2, 0, 2, 2, 0, 6, 0, 3, 3, 3],
                          [0, 0, 0, 0, 2, 2, 0, 0, 0, 3, 3, 3],
                          [0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 3, 0],
                          [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 3],
                          [1, 1, 0, 0, 0, 0, 0, 0, 3, 3, 3, 3],
                          [1, 1, 1, 0, 0, 0, 3, 3, 3, 0, 0, 3],
                          [1, 1, 1, 0, 0, 0, 3, 3, 3, 0, 0, 0],
                          [1, 1, 1, 0, 0, 0, 3, 3, 3, 0, 0, 0],
                          [1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                          [1, 0, 1, 0, 0, 0, 0, 5, 5, 0, 0, 0],
                          [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4]])
# Plot array
plt.imshow(example_array, cmap="spectral", interpolation='nearest')
A:
<code>
import numpy as np
import scipy.spatial.distance
example_array = np.array([[0, 0, 0, 2, 2, 0, 0, 0, 0, 0, 0, 0],
                          [0, 0, 2, 0, 2, 2, 0, 6, 0, 3, 3, 3],
                          [0, 0, 0, 0, 2, 2, 0, 0, 0, 3, 3, 3],
                          [0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 3, 0],
                          [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 3],
                          [1, 1, 0, 0, 0, 0, 0, 0, 3, 3, 3, 3],
                          [1, 1, 1, 0, 0, 0, 3, 3, 3, 0, 0, 3],
                          [1, 1, 1, 0, 0, 0, 3, 3, 3, 0, 0, 0],
                          [1, 1, 1, 0, 0, 0, 3, 3, 3, 0, 0, 0],
                          [1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                          [1, 0, 1, 0, 0, 0, 0, 5, 5, 0, 0, 0],
                          [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4]])
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I'm trying to use rollapply with a formula that requires 2 arguments. To my knowledge the only way (unless you create the formula from scratch) to calculate kendall tau correlation, with standard tie correction included is:
>>> import scipy
>>> x = [5.05, 6.75, 3.21, 2.66]
>>> y = [1.65, 26.5, -5.93, 7.96]
>>> z = [1.65, 2.64, 2.64, 6.95]
>>> print scipy.stats.stats.kendalltau(x, y)[0]
0.333333333333
I'm also aware of the problem with rollapply and taking two arguments, as documented here:
•	Related Question 1
•	Github Issue
•	Related Question 2
Still, I'm struggling to find a way to do the kendalltau calculation on a dataframe with multiple columns on a rolling basis.
My dataframe is something like this
A = pd.DataFrame([[1, 5, 1], [2, 4, 1], [3, 3, 1], [4, 2, 1], [5, 1, 1]],
                 columns=['A', 'B', 'C'], index = [1, 2, 3, 4, 5])
Trying to create a function that does this
In [1]:function(A, 3)  # A is df, 3 is the rolling window
Out[2]:
   A  B  C     AB     AC     BC
1  1  5  2    NaN    NaN    NaN
2  2  4  4    NaN    NaN    NaN
3  3  3  1  -1.00  -0.333   0.333
4  4  2  2  -1.00  -0.333   0.333
5  5  1  4  -1.00   1.00  -1.00
In a very preliminary approach I entertained the idea of defining the function like this:
def tau1(x):
    y = np.array(A['A']) #  keep one column fix and run it in the other two
    tau, p_value = sp.stats.kendalltau(x, y)
    return tau
 A['AB'] = pd.rolling_apply(A['B'], 3, lambda x: tau1(x))
Off course It didn't work. I got:
ValueError: all keys need to be the same shape
I understand is not a trivial problem. I appreciate any input.
A:
<code>
import pandas as pd
import numpy as np
import scipy.stats as stats
df = pd.DataFrame([[1, 5, 2], [2, 4, 4], [3, 3, 1], [4, 2, 2], [5, 1, 4]],
                 columns=['A', 'B', 'C'], index = [1, 2, 3, 4, 5])

</code>
df = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I have this example of matrix by matrix multiplication using numpy arrays:
import numpy as np
m = np.array([[1,2,3],[4,5,6],[7,8,9]])
c = np.array([0,1,2])
m * c
array([[ 0,  2,  6],
       [ 0,  5, 12],
       [ 0,  8, 18]])
How can i do the same thing if m is scipy sparse CSR matrix? The result should be csr_matrix as well.
This gives dimension mismatch:
sp.sparse.csr_matrix(m)*sp.sparse.csr_matrix(c)

A:
<code>
from scipy import sparse
import numpy as np
example_sA = sparse.csr_matrix(np.array([[1,2,3],[4,5,6],[7,8,9]]))
example_sB = sparse.csr_matrix(np.array([0,1,2]))
def f(sA = example_sA, sB = example_sB):
    # return the solution in this function
    # result = f(sA, sB)
    ### BEGIN SOLUTION


Problem:

I'm trying to reduce noise in a binary python array by removing all completely isolated single cells, i.e. setting "1" value cells to 0 if they are completely surrounded by other "0"s like this:
0 0 0
0 1 0
0 0 0
 I have been able to get a working solution by removing blobs with sizes equal to 1 using a loop, but this seems like a very inefficient solution for large arrays.
In this case, eroding and dilating my array won't work as it will also remove features with a width of 1. I feel the solution lies somewhere within the scipy.ndimage package, but so far I haven't been able to crack it. Any help would be greatly appreciated!

A:
<code>
import numpy as np
import scipy.ndimage
square = np.zeros((32, 32))
square[10:-10, 10:-10] = 1
np.random.seed(12)
x, y = (32*np.random.random((2, 20))).astype(int)
square[x, y] = 1
</code>
square = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I have two csr_matrix, c1, c2.

I want a new matrix Feature = [c1, c2]. But if I directly concatenate them horizontally this way, there's an error that says the matrix Feature is a list. How can I achieve the matrix concatenation and still get the same type of matrix, i.e. a csr_matrix?

And it doesn't work if I do this after the concatenation: Feature = csr_matrix(Feature) It gives the error:

Traceback (most recent call last):
  File "yelpfilter.py", line 91, in <module>
    Feature = csr_matrix(Feature)
  File "c:\python27\lib\site-packages\scipy\sparse\compressed.py", line 66, in __init__
    self._set_self( self.__class__(coo_matrix(arg1, dtype=dtype)) )
  File "c:\python27\lib\site-packages\scipy\sparse\coo.py", line 185, in __init__
    self.row, self.col = M.nonzero()
TypeError: __nonzero__ should return bool or int, returned numpy.bool_

A:
<code>
from scipy import sparse
c1 = sparse.csr_matrix([[0, 0, 1, 0], [2, 0, 0, 0], [0, 0, 0, 0]])
c2 = sparse.csr_matrix([[0, 3, 4, 0], [0, 0, 0, 5], [6, 7, 0, 8]])
</code>
Feature = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I have a set of data and I want to compare which line describes it best (polynomials of different orders, exponential or logarithmic).
I use Python and Numpy and for polynomial fitting there is a function polyfit().
How do I fit y = Alogx + B using polyfit()? The result should be an np.array of [A, B]
A:
<code>
import numpy as np
import scipy
x = np.array([1, 7, 20, 50, 79])
y = np.array([10, 19, 30, 35, 51])

</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I have a sparse 988x1 vector (stored in col, a column in a csr_matrix) created through scipy.sparse. Is there a way to gets its max and min value without having to convert the sparse matrix to a dense one?
numpy.max seems to only work for dense vectors.

A:
<code>
import numpy as np
from scipy.sparse import csr_matrix

np.random.seed(10)
arr = np.random.randint(4,size=(988,988))
sA = csr_matrix(arr)
col = sA.getcol(0)
</code>
Max, Min = ... # put solution in these variables
BEGIN SOLUTION
<code>


Problem:
What is the canonical way to check if a SciPy lil matrix is empty (i.e. contains only zeroes)?
I use nonzero():
def is_lil_matrix_only_zeroes(my_lil_matrix):
    return(len(my_lil_matrix.nonzero()[0]) == 0)
from scipy.sparse import csr_matrix
print(is_lil_matrix_only_zeroes(lil_matrix([[1,2,0],[0,0,3],[4,0,5]])))
print(is_lil_matrix_only_zeroes(lil_matrix([[0,0,0],[0,0,0],[0,0,0]])))
print(is_lil_matrix_only_zeroes(lil_matrix((2,3))))
print(is_lil_matrix_only_zeroes(lil_matrix([[0,0,0],[0,1,0],[0,0,0]])))
outputs
False
True
True
False
but I wonder whether there exist more direct or efficient ways, i.e. just get True or False?
A:
<code>
from scipy import sparse
sa = sparse.random(10, 10, density = 0.01, format = 'lil')
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I'd like to achieve a fourier series development for a x-y-dataset using numpy and scipy.
At first I want to fit my data with the first 8 cosines and plot additionally only the first harmonic. So I wrote the following two function defintions:
# fourier series defintions
tau = 0.045
def fourier8(x, a1, a2, a3, a4, a5, a6, a7, a8):
    return a1 * np.cos(1 * np.pi / tau * x) + \
           a2 * np.cos(2 * np.pi / tau * x) + \
           a3 * np.cos(3 * np.pi / tau * x) + \
           a4 * np.cos(4 * np.pi / tau * x) + \
           a5 * np.cos(5 * np.pi / tau * x) + \
           a6 * np.cos(6 * np.pi / tau * x) + \
           a7 * np.cos(7 * np.pi / tau * x) + \
           a8 * np.cos(8 * np.pi / tau * x)
def fourier1(x, a1):
    return a1 * np.cos(1 * np.pi / tau * x)
Then I use them to fit my data:
# import and filename
filename = 'data.txt'
import numpy as np
from scipy.optimize import curve_fit
z, Ua = np.loadtxt(filename,delimiter=',', unpack=True)
tau = 0.045
popt, pcov = curve_fit(fourier8, z, Ua)
which works as desired
But know I got stuck making it generic for arbitary orders of harmonics, e.g. I want to fit my data with the first fifteen harmonics.
How could I achieve that without defining fourier1, fourier2, fourier3 ... , fourier15?
By the way, initial guess of a1,a2,… should be set to default value.

A:
<code>
from scipy.optimize import curve_fit
import numpy as np
s = '''1.000000000000000021e-03,2.794682735905079767e+02
4.000000000000000083e-03,2.757183469104809888e+02
1.400000000000000029e-02,2.791403179603880176e+02
2.099999999999999784e-02,1.781413355804160119e+02
3.300000000000000155e-02,-2.798375517344049968e+02
4.199999999999999567e-02,-2.770513900380149721e+02
5.100000000000000366e-02,-2.713769422793179729e+02
6.900000000000000577e-02,1.280740698304900036e+02
7.799999999999999989e-02,2.800801708984579932e+02
8.999999999999999667e-02,2.790400329037249776e+02'''.replace('\n', ';')
arr = np.matrix(s)
z = np.array(arr[:, 0]).squeeze()
Ua = np.array(arr[:, 1]).squeeze()
tau = 0.045
degree = 15
</code>
popt, pcov = ... # put solution in these variables
BEGIN SOLUTION
<code>


Problem:
I have a set of data and I want to compare which line describes it best (polynomials of different orders, exponential or logarithmic).
I use Python and Numpy and for polynomial fitting there is a function polyfit(). But I found no such functions for exponential and logarithmic fitting.
How do I fit y = A*exp(Bx) + C ? The result should be an np.array of [A, B, C]. I know that polyfit performs bad for this function, so I would like to use curve_fit to solve the problem, and it should start from initial guess p0.
A:
<code>
import numpy as np
import scipy.optimize
y = np.array([1, 7, 20, 50, 79])
x = np.array([10, 19, 30, 35, 51])
p0 = (4, 0.1, 1)
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:

Suppose I have a integer matrix which represents who has emailed whom and how many times. I want to find people that have not emailed each other. For social network analysis I'd like to make a simple undirected graph. So I need to convert the matrix to binary matrix.
My question: is there a fast, convenient way to reduce the decimal matrix to a binary matrix.
Such that:
26, 3, 0
3, 195, 1
0, 1, 17
Becomes:
0, 0, 1
0, 0, 0
1, 0, 0

A:

<code>
import scipy
import numpy as np
a = np.array([[26, 3, 0], [3, 195, 1], [0, 1, 17]])
</code>
a = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
After clustering a distance matrix with scipy.cluster.hierarchy.linkage, and assigning each sample to a cluster using scipy.cluster.hierarchy.cut_tree, I would like to extract one element out of each cluster, which is the k-th closest to that cluster's centroid.
•	I would be the happiest if an off-the-shelf function existed for this, but in the lack thereof:
•	some suggestions were already proposed here for extracting the centroids themselves, but not the closest-to-centroid elements.
•	Note that this is not to be confused with the centroid linkage rule in scipy.cluster.hierarchy.linkage. I have already carried out the clustering itself, just want to access the closest-to-centroid elements.
What I want is the index of the k-closest element in original data for each cluster, i.e., result[0] is the index of the k-th closest element to centroid of cluster 0.
A:
<code>
import numpy as np
import scipy.spatial
centroids = np.random.rand(5, 3)
data = np.random.rand(100, 3)
k = 3
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I have a binary array, say, a = np.random.binomial(n=1, p=1/2, size=(9, 9)). I perform median filtering on it using a 3 x 3 kernel on it, like say, b = nd.median_filter(a, 3). I would expect that this should perform median filter based on the pixel and its eight neighbours. However, I am not sure about the placement of the kernel. The documentation says,

origin : scalar, optional.
The origin parameter controls the placement of the filter. Default 0.0.

Now, I want to shift this filter one cell to the right.How can I achieve it?
Thanks.

A:
<code>
import numpy as np
import scipy.ndimage

a= np.zeros((5, 5))
a[1:4, 1:4] = np.arange(3*3).reshape((3, 3))
</code>
b = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I have problems using scipy.sparse.csr_matrix:
for instance:
a = csr_matrix([[1,2,3],[4,5,6]])
b = csr_matrix([[7,8,9],[10,11,12]])
how to merge them into
[[1,2,3,7,8,9],[4,5,6,10,11,12]]
I know a way is to transfer them into numpy array first:
csr_matrix(numpy.hstack((a.toarray(),b.toarray())))
but it won't work when the matrix is huge and sparse, because the memory would run out.
so are there any way to merge them together in csr_matrix?
any answers are appreciated!
A:
<code>
from scipy import sparse
sa = sparse.random(10, 10, density = 0.01, format = 'csr')
sb = sparse.random(10, 10, density = 0.01, format = 'csr')
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I have two csr_matrix, c1 and c2.

I want a new sparse matrix Feature = [c1, c2], that is, to stack c1 and c2 horizontally to get a new sparse matrix.

To make use of sparse matrix's memory efficiency, I don't want results as dense arrays.

But if I directly concatenate them this way, there's an error that says the matrix Feature is a list.

And if I try this: Feature = csr_matrix(Feature) It gives the error:

Traceback (most recent call last):
  File "yelpfilter.py", line 91, in <module>
    Feature = csr_matrix(Feature)
  File "c:\python27\lib\site-packages\scipy\sparse\compressed.py", line 66, in __init__
    self._set_self( self.__class__(coo_matrix(arg1, dtype=dtype)) )
  File "c:\python27\lib\site-packages\scipy\sparse\coo.py", line 185, in __init__
    self.row, self.col = M.nonzero()
TypeError: __nonzero__ should return bool or int, returned numpy.bool_

Any help would be appreciated!

A:
<code>
from scipy import sparse
c1 = sparse.csr_matrix([[0, 0, 1, 0], [2, 0, 0, 0], [0, 0, 0, 0]])
c2 = sparse.csr_matrix([[0, 3, 4, 0], [0, 0, 0, 5], [6, 7, 0, 8]])
</code>
Feature = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
Give the N and P, I want to get a 2D binomial distribution probability matrix M,
for i in range(N+1):
   for j in range(i+1):
      M[i,j] = choose(i, j) * p**j * (1-p)**(i-j)
other value = 0

I want to know is there any fast way to get this matrix, instead of the for loop. the N may be bigger than 100,000

A:
<code>
import numpy as np
import scipy.stats
N = 3
p = 0.5
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
How do we pass two datasets in scipy.stats.anderson_ksamp?

The anderson function asks only for one parameter and that should be 1-d array. So I am wondering how to pass two different arrays to be compared in it?
Further, I want to interpret the result, that is, telling whether the two different arrays are drawn from the same population at the 5% significance level, result should be `True` or `False` .
A:
<code>
import numpy as np
import scipy.stats as ss
x1=[38.7,  41.5,  43.8,  44.5,  45.5,  46.0,  47.7,  58.0]
x2=[39.2,  39.3,  39.7,  41.4,  41.8,  42.9,  43.3,  45.8]
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
Given two sets of points in n-dimensional space, how can one map points from one set to the other, such that each point is only used once and the total euclidean distance between the pairs of points is minimized?
For example,
import matplotlib.pyplot as plt
import numpy as np
# create six points in 2d space; the first three belong to set "A" and the
# second three belong to set "B"
x = [1, 2, 3, 1.8, 1.9, 3.4]
y = [2, 3, 1, 2.6, 3.4, 0.4]
colors = ['red'] * 3 + ['blue'] * 3
plt.scatter(x, y, c=colors)
plt.show()
So in the example above, the goal would be to map each red point to a blue point such that each blue point is only used once and the sum of the distances between points is minimized.
The application I have in mind involves a fairly small number of datapoints in 3-dimensional space, so the brute force approach might be fine, but I thought I would check to see if anyone knows of a more efficient or elegant solution first.
The result should be an assignment of points in second set to corresponding elements in the first set.
For example, a matching solution is
Points1 <-> Points2
    0   ---     2
    1   ---     0
    2   ---     1
and the result is [2, 0, 1]

A:
<code>
import numpy as np
import scipy.spatial
import scipy.optimize
points1 = np.array([(x, y) for x in np.linspace(-1,1,7) for y in np.linspace(-1,1,7)])
N = points1.shape[0]
points2 = 2*np.random.rand(N,2)-1
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I am trying to optimise a function using the fminbound function of the scipy.optimize module. I want to set parameter bounds to keep the answer physically sensible (e.g. > 0).
import scipy.optimize as sciopt
import numpy as np
The arrays:
x = np.array([[ 1247.04,  1274.9 ,  1277.81,  1259.51,  1246.06,  1230.2 ,
     1207.37,  1192.  ,  1180.84,  1182.76,  1194.76,  1222.65],
   [  589.  ,   581.29,   576.1 ,   570.28,   566.45,   575.99,
      601.1 ,   620.6 ,   637.04,   631.68,   611.79,   599.19]])
y = np.array([ 1872.81,  1875.41,  1871.43,  1865.94,  1854.8 ,  1839.2 ,
    1827.82,  1831.73,  1846.68,  1856.56,  1861.02,  1867.15])
I managed to optimise the linear function within the parameter bounds when I use only one parameter:
fp   = lambda p, x: x[0]+p*x[1]
e    = lambda p, x, y: ((fp(p,x)-y)**2).sum()
pmin = 0.5 # mimimum bound
pmax = 1.5 # maximum bound
popt = sciopt.fminbound(e, pmin, pmax, args=(x,y))
This results in popt = 1.05501927245
However, when trying to optimise with multiple parameters, I get the following error message:
fp   = lambda p, x: p[0]*x[0]+p[1]*x[1]
e    = lambda p, x, y: ((fp(p,x)-y)**2).sum()
pmin = np.array([0.5,0.5]) # mimimum bounds
pmax = np.array([1.5,1.5]) # maximum bounds
popt = sciopt.fminbound(e, pmin, pmax, args=(x,y))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/scipy/optimize/optimize.py", line 949, in fminbound
    if x1 > x2:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
I have tried to vectorize e (np.vectorize) but the error message remains the same. I understand that fminbound expects a float or array scalar as bounds. Is there another function that would work for this problem? The result should be solutions for p[0] and p[1] that minimize the objective function.

A:
<code>
import numpy as np
import scipy.optimize as sciopt
x = np.array([[ 1247.04,  1274.9 ,  1277.81,  1259.51,  1246.06,  1230.2 ,
     1207.37,  1192.  ,  1180.84,  1182.76,  1194.76,  1222.65],
   [  589.  ,   581.29,   576.1 ,   570.28,   566.45,   575.99,
      601.1 ,   620.6 ,   637.04,   631.68,   611.79,   599.19]])
y = np.array([ 1872.81,  1875.41,  1871.43,  1865.94,  1854.8 ,  1839.2 ,
    1827.82,  1831.73,  1846.68,  1856.56,  1861.02,  1867.15])
fp   = lambda p, x: p[0]*x[0]+p[1]*x[1]
e    = lambda p, x, y: ((fp(p,x)-y)**2).sum()
pmin = np.array([0.5,0.7]) # mimimum bounds
pmax = np.array([1.5,1.8]) # maximum bounds
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
How to find relative extrema of a 2D array? An element is a relative extrema if it is less or equal to the neighbouring n (e.g. n = 2) elements forwards and backwards in the row.
The result should be a list of indices of those elements, [0, 1] stands for arr[0][1]. It should be arranged like
[[0, 1], [0, 5], [1, 1], [1, 4], [2, 3], [2, 5], ...]
A:
<code>
import numpy as np
from scipy import signal
arr = np.array([[-624.59309896, -624.59309896, -624.59309896,
                      -625., -625., -625.,], [3, 0, 0, 1, 2, 4]])
n = 2
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
After clustering a distance matrix with scipy.cluster.hierarchy.linkage, and assigning each sample to a cluster using scipy.cluster.hierarchy.cut_tree, I would like to extract one element out of each cluster, which is the closest to that cluster's centroid.
•	I would be the happiest if an off-the-shelf function existed for this, but in the lack thereof:
•	some suggestions were already proposed here for extracting the centroids themselves, but not the closest-to-centroid elements.
•	Note that this is not to be confused with the centroid linkage rule in scipy.cluster.hierarchy.linkage. I have already carried out the clustering itself, just want to access the closest-to-centroid elements.
What I want is the vector of the closest point to each cluster, i.e., result[0] is the vector of the closest element to cluster 0.
A:
<code>
import numpy as np
import scipy.spatial
centroids = np.random.rand(5, 3)
data = np.random.rand(100, 3)
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I am having a problem with minimization procedure. Actually, I could not create a correct objective function for my problem.
Problem definition
•	My function: yn = a_11*x1**2 + a_12*x2**2 + ... + a_m*xn**2,where xn- unknowns, a_m - coefficients. n = 1..N, m = 1..M
•	In my case, N=5 for x1,..,x5 and M=3 for y1, y2, y3.
I need to find the optimum: x1, x2,...,x5 so that it can satisfy the y
My question:
•	How to solve the question using scipy.optimize?
My code:   (tried in lmfit, but return errors. Therefore I would ask for scipy solution)
import numpy as np
from lmfit import Parameters, minimize
def func(x,a):
    return np.dot(a, x**2)
def residual(pars, a, y):
    vals = pars.valuesdict()
    x = vals['x']
    model = func(x,a)
    return (y - model) **2
def main():
    # simple one: a(M,N) = a(3,5)
    a = np.array([ [ 0, 0, 1, 1, 1 ],
                   [ 1, 0, 1, 0, 1 ],
                   [ 0, 1, 0, 1, 0 ] ])
    # true values of x
    x_true = np.array([10, 13, 5, 8, 40])
    # data without noise
    y = func(x_true,a)
    #************************************
    # Apriori x0
    x0 = np.array([2, 3, 1, 4, 20])
    fit_params = Parameters()
    fit_params.add('x', value=x0)
    out = minimize(residual, fit_params, args=(a, y))
    print out
if __name__ == '__main__':
main()
Result should be optimal x array.

A:
<code>
import scipy.optimize
import numpy as np
np.random.seed(42)
a = np.random.rand(3,5)
x_true = np.array([10, 13, 5, 8, 40])
y = a.dot(x_true ** 2)
x0 = np.array([2, 3, 1, 4, 20])
</code>
out = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I think my questions has something in common with this question or others, but anyway, mine is not specifically about them.
I would like, after having found the voronoi tessallination for certain points, be able to check where other given points sit within the tessellination. In particular:
Given say 50 extra-points, I want to be able to count how many of these extra points each voronoi cell contains.
My MWE
from scipy.spatial import ConvexHull, Voronoi
points = [[0,0], [1,4], [2,3], [4,1], [1,1], [2,2], [5,3]]
#voronoi
vor = Voronoi(points)
Now I am given extra points
extraPoints = [[0.5,0.2], [3, 0], [4,0],[5,0], [4,3]]
# In this case we have that the first point is in the bottom left,
# the successive three are in the bottom right and the last one
# is in the top right cell.
I was thinking to use the fact that you can get vor.regions or vor.vertices, however I really couldn't come up with anything..
Is there parameter or a way to make this? The result I want is an np.array containing indices standing for regions occupied by different points, and that should be defined by Voronoi cell.
A:
<code>
import scipy.spatial
points = [[0,0], [1,4], [2,3], [4,1], [1,1], [2,2], [5,3]]
vor = scipy.spatial.Voronoi(points)
extraPoints = [[0.5,0.2], [3, 0], [4,0],[5,0], [4,3]]
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
After clustering a distance matrix with scipy.cluster.hierarchy.linkage, and assigning each sample to a cluster using scipy.cluster.hierarchy.cut_tree, I would like to extract one element out of each cluster, which is the closest to that cluster's centroid.
•	I would be the happiest if an off-the-shelf function existed for this, but in the lack thereof:
•	some suggestions were already proposed here for extracting the centroids themselves, but not the closest-to-centroid elements.
•	Note that this is not to be confused with the centroid linkage rule in scipy.cluster.hierarchy.linkage. I have already carried out the clustering itself, just want to access the closest-to-centroid elements.
What I want is the index of the closest element in original data for each cluster, i.e., result[0] is the index of the closest element to cluster 0.
A:
<code>
import numpy as np
import scipy.spatial
centroids = np.random.rand(5, 3)
data = np.random.rand(100, 3)
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I have a table of measured values for a quantity that depends on two parameters. So say I have a function fuelConsumption(speed, temperature), for which data on a mesh are known.
Now I want to interpolate the expected fuelConsumption for a lot of measured data points (speed, temperature) from a pandas.DataFrame (and return a vector with the values for each data point).
I am currently using SciPy's interpolate.interp2d for cubic interpolation, but when passing the parameters as two vectors [s1,s2] and [t1,t2] (only two ordered values for simplicity) it will construct a mesh and return:
[[f(s1,t1), f(s2,t1)], [f(s1,t2), f(s2,t2)]]
The result I am hoping to get is:
[f(s1,t1), f(s2, t2)]
How can I interpolate to get the output I want?
I want to use function interpolated on x, y, z to compute values on arrays s and t, and the result should be like mentioned above.
A:
<code>
import numpy as np
import scipy.interpolate
s = np.linspace(-1, 1, 50)
t = np.linspace(-2, 0, 50)
x, y = np.ogrid[-1:1:10j,-2:0:10j]
z = (x + y)*np.exp(-6.0 * (x * x + y * y))
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I think my questions has something in common with this question or others, but anyway, mine is not specifically about them.
I would like, after having found the voronoi tessallination for certain points, be able to check where other given points sit within the tessellination. In particular:
Given say 50 extra-points, I want to be able to count how many of these extra points each voronoi cell contains.
My MWE
from scipy.spatial import ConvexHull, Voronoi
points = [[0,0], [1,4], [2,3], [4,1], [1,1], [2,2], [5,3]]
#voronoi
vor = Voronoi(points)
Now I am given extra points
extraPoints = [[0.5,0.2], [3, 0], [4,0],[5,0], [4,3]]
# In this case we have that the first point is in the bottom left,
# the successive three are in the bottom right and the last one
# is in the top right cell.
I was thinking to use the fact that you can get vor.regions or vor.vertices, however I really couldn't come up with anything..
Is there parameter or a way to make this? The result I want is an np.array containing indices standing for regions occupied by different points, i.e., 1 for [1, 4]’s region.
A:
<code>
import scipy.spatial
points = [[0,0], [1,4], [2,3], [4,1], [1,1], [2,2], [5,3]]
vor = scipy.spatial.Voronoi(points)
extraPoints = [[0.5,0.2], [3, 0], [4,0],[5,0], [4,3]]
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I'm trying to solve a simple ODE to visualise the temporal response, which works well for constant input conditions using the new solve_ivp integration API in SciPy. For example:
def dN1_dt_simple(t, N1):
    return -100 * N1
sol = solve_ivp(fun=dN1_dt_simple, t_span=time_span, y0=[N0,])
However, I wonder is it possible to plot the response to a time-varying input? For instance, rather than having y0 fixed at N0, can I find the response to a simple sinusoid? Specifically, I want to change dy/dt = -100*y + sin(t) to let it become time-variant. The result I want is values of solution at time points.
Is there a compatible way to pass time-varying input conditions into the API?
A:
<code>
import scipy.integrate
import numpy as np
N0 = 10
time_span = [-0.1, 0.1]
</code>
solve this question with example variable `sol` and set `result = sol.y`
BEGIN SOLUTION
<code>


Problem:
I have an array which I want to interpolate over the 1st axes. At the moment I am doing it like this example:
import numpy as np
from scipy.interpolate import interp1d
array = np.random.randint(0, 9, size=(100, 100, 100))
new_array = np.zeros((1000, 100, 100))
x = np.arange(0, 100, 1)
x_new = np.arange(0, 100, 0.1)
for i in x:
    for j in x:
        f = interp1d(x, array[:, i, j])
        new_array[:, i, j] = f(xnew)
The data I use represents 10 years of 5-day averaged values for each latitude and longitude in a domain. I want to create an array of daily values.
I have also tried using splines. I don't really know how they work but it was not much faster.
Is there a way to do this without using for loops? The result I want is an np.array of transformed x_new values using interpolated function.
Thank you in advance for any suggestions.
A:
<code>
import numpy as np
import scipy.interpolate
array = np.random.randint(0, 9, size=(10, 10, 10))
x = np.linspace(0, 10, 10)
x_new = np.linspace(0, 10, 100)
</code>
new_array = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:

Suppose I have a integer matrix which represents who has emailed whom and how many times. For social network analysis I'd like to make a simple undirected graph. So I need to convert the matrix to binary matrix.
My question: is there a fast, convenient way to reduce the decimal matrix to a binary matrix.
Such that:
26, 3, 0
3, 195, 1
0, 1, 17
Becomes:
1, 1, 0
1, 1, 1
0, 1, 1

A:

<code>
import scipy
import numpy as np
a = np.array([[26, 3, 0], [3, 195, 1], [0, 1, 17]])
</code>
a = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I have some data that comes in the form (x, y, z, V) where x,y,z are distances, and V is the moisture. I read a lot on StackOverflow about interpolation by python like this and this valuable posts, but all of them were about regular grids of x, y, z. i.e. every value of x contributes equally with every point of y, and every point of z. On the other hand, my points came from 3D finite element grid (as below), where the grid is not regular.
The two mentioned posts 1 and 2, defined each of x, y, z as a separate numpy array then they used something like cartcoord = zip(x, y) then scipy.interpolate.LinearNDInterpolator(cartcoord, z) (in a 3D example). I can not do the same as my 3D grid is not regular, thus not each point has a contribution to other points, so if when I repeated these approaches I found many null values, and I got many errors.
Here are 10 sample points in the form of [x, y, z, V]
data = [[27.827, 18.530, -30.417, 0.205] , [24.002, 17.759, -24.782, 0.197] ,
[22.145, 13.687, -33.282, 0.204] , [17.627, 18.224, -25.197, 0.197] ,
[29.018, 18.841, -38.761, 0.212] , [24.834, 20.538, -33.012, 0.208] ,
[26.232, 22.327, -27.735, 0.204] , [23.017, 23.037, -29.230, 0.205] ,
[28.761, 21.565, -31.586, 0.211] , [26.263, 23.686, -32.766, 0.215]]

I want to get the interpolated value V of the point (25, 20, -30) and (27, 20, -32) as a list.
How can I get it?

A:
<code>
import numpy as np
import scipy.interpolate

points = np.array([
        [ 27.827,  18.53 , -30.417], [ 24.002,  17.759, -24.782],
        [ 22.145,  13.687, -33.282], [ 17.627,  18.224, -25.197],
        [ 29.018,  18.841, -38.761], [ 24.834,  20.538, -33.012],
        [ 26.232,  22.327, -27.735], [ 23.017,  23.037, -29.23 ],
        [ 28.761,  21.565, -31.586], [ 26.263,  23.686, -32.766]])
V = np.array([0.205,  0.197,  0.204,  0.197,  0.212,
                   0.208,  0.204,  0.205, 0.211,  0.215])
request = np.array([[25, 20, -30], [27, 20, -32]])
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
How does one convert a list of Z-scores from the Z-distribution (standard normal distribution, Gaussian distribution) to left-tailed p-values? I have yet to find the magical function in Scipy's stats module to do this, but one must be there.
A:
<code>
import numpy as np
import scipy.stats
z_scores = np.array([-3, -2, 0, 2, 2.5])
</code>
p_values = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I’m trying to solve a simple ODE to visualise the temporal response, which works well for constant input conditions using the new solve_ivp integration API in SciPy. For example:
def dN1_dt_simple(t, N1):
    return -100 * N1
sol = solve_ivp(fun=dN1_dt_simple, t_span=time_span, y0=[N0,])
However, I wonder is it possible to plot the response to a time-varying input? For instance, rather than having y0 fixed at N0, can I find the response to a simple sinusoid? Specifically, I want to add `-cos(t)` to original y. The result I want is values of solution at time points.
Is there a compatible way to pass time-varying input conditions into the API?
A:
<code>
import scipy.integrate
import numpy as np
N0 = 10
time_span = [-0.1, 0.1]
</code>
solve this question with example variable `sol` and set `result = sol.y`
BEGIN SOLUTION
<code>


Problem:
I have a list of numpy vectors of the format:
    [array([[-0.36314615,  0.80562619, -0.82777381, ...,  2.00876354,2.08571887, -1.24526026]]),
     array([[ 0.9766923 , -0.05725135, -0.38505339, ...,  0.12187988,-0.83129255,  0.32003683]]),
     array([[-0.59539878,  2.27166874,  0.39192573, ..., -0.73741573,1.49082653,  1.42466276]])]

here, only 3 vectors in the list are shown. I have 100s..
The maximum number of elements in one vector is around 10 million
All the arrays in the list have unequal number of elements but the maximum number of elements is fixed.
Is it possible to create a sparse matrix using these vectors in python such that I have padded zeros to the end of elements for the vectors which are smaller than the maximum size?

A:
<code>
import numpy as np
import scipy.sparse as sparse

np.random.seed(10)
max_vector_size = 1000
vectors = [np.random.randint(100,size=900),np.random.randint(100,size=max_vector_size),np.random.randint(100,size=950)]
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I'm trying to create a 2-dimensional array in Scipy/Numpy where each value represents the euclidean distance from the center. It's supposed to have the same shape as the first two dimensions of a 3-dimensional array (an image, created via scipy.misc.fromimage).
I'm very new to Scipy, and would like to know if there's a more elegant, idiomatic way of doing the same thing. I found the scipy.spatial.distance.cdist function, which seems promising, but I'm at a loss regarding how to fit it into this problem.
def get_distance_2(y, x):
    mid = ...  # needs to be a array of the shape (rows, cols, 2)?
    return scipy.spatial.distance.cdist(scipy.dstack((y, x)), mid)
Just to clarify, what I'm looking for is something like this (for a 6 x 6 array). That is, to compute (Euclidean) distances from center point to every point in the image.
[[ 3.53553391  2.91547595  2.54950976  2.54950976  2.91547595  3.53553391]
 [ 2.91547595  2.12132034  1.58113883  1.58113883  2.12132034  2.91547595]
 [ 2.54950976  1.58113883  0.70710678  0.70710678  1.58113883  2.54950976]
 [ 2.54950976  1.58113883  0.70710678  0.70710678  1.58113883  2.54950976]
 [ 2.91547595  2.12132034  1.58113883  1.58113883  2.12132034  2.91547595]
 [ 3.53553391  2.91547595  2.54950976  2.54950976  2.91547595  3.53553391]]
A:
<code>
import numpy as np
from scipy.spatial import distance
def f(shape = (6, 6)):
    # return the solution in this function
    # result = f(shape = (6, 6))
    ### BEGIN SOLUTION


Problem:
Having difficulty generating a tridiagonal matrix from numpy arrays. I managed to replicate the results given here, but I'm not able to apply these techniques to my problem. I may also be misunderstanding the application of scipy.sparse.diag.
For context, I'm working on a problem which requires the generation of a tridiagonal matrix to solve an ordinary differential equation numerically using finite differences.
from scipy.sparse import diags
import numpy as np
v1 = [3*i**2 +(i/2) for i in range(1, 6)]
v2 = [-(6*i**2 - 1) for i in range(1, 6)]
v3 = [3*i**2 -(i/2) for i in range(1, 6)]
matrix = np.array([v1, v2, v3])
matrix is equal to.
array([[3.5,   13. ,   28.5,   50. ,   77.5],
       [-5. ,  -23. ,  -53. ,  -95. , -149. ],
       [2.5,   11. ,   25.5,   46. ,   72.5]])
After working through the Scipy documentation and the examples in the link above, I was expecting the following code to yield Tridiagonal_1, but instead get Tridiagonal_2.
diags(matrix, [-1,0,1], (5, 5)).toarray()
expected Tridiagonal_1:
array([[  -5. ,    2.5 ,     0. ,    0. ,     0. ],
       [  13. ,   -23. ,    11. ,    0. ,     0. ],
       [   0. ,    28.5.,  -53. ,   25.5,     0. ],
       [   0. ,    0. ,     50 ,   -95.,     46. ],
       [   0. ,    0. ,      0. ,   77.5., -149. ]])
Code yielded Tridiagonal_2:
array([[  -5. ,    2.5,    0. ,    0. ,    0. ],
       [   3.5,  -23. ,   11. ,    0. ,    0. ],
       [   0. ,   13. ,  -53. ,   25.5,    0. ],
       [   0. ,    0. ,   28.5,  -95. ,   46. ],
       [   0. ,    0. ,    0. ,   50. , -149. ]])
I was expecting offset = [-1,0,1] to shift the diagonal entries to the left, but the first offset is shifting the first diag to the next row. Is this correct or is there an error in my code causing this behaviour?
A:
<code>
from scipy import sparse
import numpy as np
matrix = np.array([[3.5,   13. ,   28.5,   50. ,   77.5],
                   [-5. ,  -23. ,  -53. ,  -95. , -149. ],
                   [2.5,   11. ,   25.5,   46. ,   72.5]])
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I'm using scipy.optimize.minimize to solve a complex reservoir optimization model (SQSLP and COBYLA as the problem is constrained by both bounds and constraint equations). There is one decision variable per day (storage), and releases from the reservoir are calculated as a function of change in storage, within the objective function. Penalties based on releases and storage penalties are then applied with the goal of minimizing penalties (the objective function is a summation of all penalties). I've added some constraints within this model to limit the change in storage to the physical system limits which is the difference between decision variable x(t+1) and x(t), and also depends on inflows at that time step I(t). These constraints are added to the list of constraint dictionaries using a for loop. Constraints added outside of this for loop function as they should. However the constraints involving time that are initiated within the for loop, do not.
Obviously the problem is complex so I've recreated a simpler version to illustrate the problem. This problem has four decision variables and seeks to minimize the objective function (which I've called function) with constraints of steady state (I = inflow must equal x = outflow) and non negativity (ie. outflows x cannot be negative):
    import numpy as np
    from scipy.optimize import minimize
    def function(x):
        return -1*(18*x[0]+16*x[1]+12*x[2]+11*x[3])
    I=np.array((20,50,50,80))
    x0=I
    cons=[]
    steadystate={'type':'eq', 'fun': lambda x: x.sum()-I.sum() }
    cons.append(steadystate)
    for t in range (4):
        def const(x):
            y=x[t]
            return y
        cons.append({'type':'ineq', 'fun': const})
    out=minimize(function, x0, method="SLSQP", constraints=cons)
    x=out["x"]
The constraints initiated in the for loop are non-negativity constraints but the optimization gives negative values for the decision variables. It does adhere to the steadystate constraint, however.
Any ideas where I'm going wrong? I've seen constraints initiated similarly in other applications so I can't figure it out but assume it's something simple. I have hundreds of constraints to initiate in my full-scale version of this code so writing them out as in the second example will not be ideal.
A:
<code>
import numpy as np
from scipy.optimize import minimize

def function(x):
    return -1*(18*x[0]+16*x[1]+12*x[2]+11*x[3])

I=np.array((20,50,50,80))
x0=I

cons=[]
steadystate={'type':'eq', 'fun': lambda x: x.sum()-I.sum() }
cons.append(steadystate)
</code>
Carefully set `cons` for running the following code.
BEGIN SOLUTION
<code>


Problem:
I want to capture an integral of a column of my dataframe with a time index. This works fine for a grouping that happens every time interval.
from scipy import integrate
>>> df
Time                      A
2017-12-18 19:54:40   -50187.0
2017-12-18 19:54:45   -60890.5
2017-12-18 19:54:50   -28258.5
2017-12-18 19:54:55    -8151.0
2017-12-18 19:55:00    -9108.5
2017-12-18 19:55:05   -12047.0
2017-12-18 19:55:10   -19418.0
2017-12-18 19:55:15   -50686.0
2017-12-18 19:55:20   -57159.0
2017-12-18 19:55:25   -42847.0
>>> integral_df = df.groupby(pd.Grouper(freq='25S')).apply(integrate.trapz)
Time                       A
2017-12-18 19:54:35   -118318.00
2017-12-18 19:55:00   -115284.75
2017-12-18 19:55:25         0.00
Freq: 25S, Name: A, dtype: float64
EDIT:
The scipy integral function automatically uses the time index to calculate it's result.
This is not true. You have to explicitly pass the conversion to np datetime in order for scipy.integrate.trapz to properly integrate using time. See my comment on this question.
But, i'd like to take a rolling integral instead. I've tried Using rolling functions found on SO, But the code was getting messy as I tried to workout my input to the integrate function, as these rolling functions don't return dataframes.
How can I take a rolling integral over time over a function of one of my dataframe columns?
A:
<code>
import pandas as pd
import io
from scipy import integrate
string = '''
Time                      A
2017-12-18-19:54:40   -50187.0
2017-12-18-19:54:45   -60890.5
2017-12-18-19:54:50   -28258.5
2017-12-18-19:54:55    -8151.0
2017-12-18-19:55:00    -9108.5
2017-12-18-19:55:05   -12047.0
2017-12-18-19:55:10   -19418.0
2017-12-18-19:55:15   -50686.0
2017-12-18-19:55:20   -57159.0
2017-12-18-19:55:25   -42847.0
'''
df = pd.read_csv(io.StringIO(string), sep = '\s+')
</code>
integral_df = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
I have the following code to run Wilcoxon rank-sum test
print stats.ranksums(pre_course_scores, during_course_scores)
RanksumsResult(statistic=8.1341352369246582, pvalue=4.1488919597127145e-16)

However, I am interested in extracting the pvalue from the result. I could not find a tutorial about this. i.e.Given two ndarrays, pre_course_scores, during_course_scores, I want to know the pvalue of ranksum. Can someone help?

A:
<code>
import numpy as np
from scipy import stats
np.random.seed(10)
pre_course_scores = np.random.randn(10)
during_course_scores = np.random.randn(10)
</code>
p_value = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
First off, I'm no mathmatician. I admit that. Yet I still need to understand how ScyPy's sparse matrices work arithmetically in order to switch from a dense NumPy matrix to a SciPy sparse matrix in an application I have to work on. The issue is memory usage. A large dense matrix will consume tons of memory.
The formula portion at issue is where a matrix is added to some scalars.
A = V + x
B = A + y
Where V is a square sparse matrix (its large, say 60,000 x 60,000).
What I want is that x, y will only be added to non-zero values in V.
With a SciPy, not all sparse matrices support the same features, like scalar addition. dok_matrix (Dictionary of Keys) supports scalar addition, but it looks like (in practice) that it's allocating each matrix entry, effectively rendering my sparse dok_matrix as a dense matrix with more overhead. (not good)
The other matrix types (CSR, CSC, LIL) don't support scalar addition.
I could try constructing a full matrix with the scalar value x, then adding that to V. I would have no problems with matrix types as they all seem to support matrix addition. However I would have to eat up a lot of memory to construct x as a matrix, and the result of the addition could end up being fully populated matrix as well.
There must be an alternative way to do this that doesn't require allocating 100% of a sparse matrix. I’d like to solve the problem on coo matrix first.
I'm will to accept that large amounts of memory are needed, but I thought I would seek some advice first. Thanks.
A:
<code>
from scipy import sparse
V = sparse.random(10, 10, density = 0.05, format = 'coo', random_state = 42)
x = 100
y = 99
</code>
V = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
Is there a simple and efficient way to make a sparse scipy matrix (e.g. lil_matrix, or csr_matrix) symmetric?
Currently I have a lil sparse matrix, and not both of sA[i,j] and sA[j,i] have element for any i,j.
When populating a large sparse co-occurrence matrix it would be highly inefficient to fill in [row, col] and [col, row] at the same time. What I'd like to be doing is:
for i in data:
    for j in data:
        if have_element(i, j):
            lil_sparse_matrix[i, j] = some_value
            # want to avoid this:
            # lil_sparse_matrix[j, i] = some_value
# this is what I'm looking for:
lil_sparse.make_symmetric()
and it let sA[i,j] = sA[j,i] for any i, j.

This is similar to stackoverflow's numpy-smart-symmetric-matrix question, but is particularly for scipy sparse matrices.

A:

import numpy as np
from scipy.sparse import lil_matrix
example_sA = sparse.random(10, 10, density=0.1, format='lil')
def f(sA = example_sA):
    # return the solution in this function
    # sA = f(sA)
    ### BEGIN SOLUTION


Problem:
I have the following code to run Wilcoxon rank-sum test
print stats.ranksums(pre_course_scores, during_course_scores)
RanksumsResult(statistic=8.1341352369246582, pvalue=4.1488919597127145e-16)

However, I am interested in extracting the pvalue from the result. I could not find a tutorial about this. i.e.Given two ndarrays, pre_course_scores, during_course_scores, I want to know the pvalue of ranksum. Can someone help?

A:
<code>
import numpy as np
from scipy import stats
example_pre_course_scores = np.random.randn(10)
example_during_course_scores = np.random.randn(10)
def f(pre_course_scores = example_pre_course_scores, during_course_scores = example_during_course_scores):
    # return the solution in this function
    # p_value = f(pre_course_scores, during_course_scores)
    ### BEGIN SOLUTION


Problem:
I have problems using scipy.sparse.csr_matrix:
for instance:
a = csr_matrix([[1,2,3],[4,5,6]])
b = csr_matrix([[7,8,9],[10,11,12]])
how to merge them into
[[1,2,3],[4,5,6],[7,8,9],[10,11,12]]
I know a way is to transfer them into numpy array first:
csr_matrix(numpy.vstack((a.toarray(),b.toarray())))
but it won't work when the matrix is huge and sparse, because the memory would run out.
so are there any way to merge them together in csr_matrix?
any answers are appreciated!
A:
<code>
from scipy import sparse
sa = sparse.random(10, 10, density = 0.01, format = 'csr')
sb = sparse.random(10, 10, density = 0.01, format = 'csr')
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:

I'm trying to reduce noise in a python image array by removing all completely isolated single cells, i.e. setting nonzero value cells to 0 if they are completely surrounded by other "0"s like this:
0 0 0
0 8 0
0 0 0
 I have been able to get a working solution by removing blobs with sizes equal to 1 using a loop, but this seems like a very inefficient solution for large arrays.
In this case, eroding and dilating my array won't work as it will also remove features with a width of 1. I feel the solution lies somewhere within the scipy.ndimage package, but so far I haven't been able to crack it. Any help would be greatly appreciated!

A:
<code>
import numpy as np
import scipy.ndimage
square = np.zeros((32, 32))
square[10:-10, 10:-10] = np.random.randint(1, 255, size = (12, 12))
np.random.seed(12)
x, y = (32*np.random.random((2, 20))).astype(int)
square[x, y] = np.random.randint(1, 255, size = (20,))

</code>
square = ... # put solution in this variable
BEGIN SOLUTION
<code>


Problem:
How do we pass four datasets in scipy.stats.anderson_ksamp?

The anderson function asks only for one parameter and that should be 1-d array. So I am wondering how to pass four different arrays to be compared in it? Thanks
A:
<code>
import numpy as np
import scipy.stats as ss
x1=[38.7,  41.5,  43.8,  44.5,  45.5,  46.0,  47.7,  58.0]
x2=[39.2,  39.3,  39.7,  41.4,  41.8,  42.9,  43.3,  45.8]
x3=[34.0,  35.0,  39.0,  40.0,  43.0,  43.0,  44.0,  45.0]
x4=[34.0,  34.8,  34.8,  35.4,  37.2,  37.8,  41.2,  42.8]
</code>
statistic, critical_values, significance_level = ... # put solution in these variables
BEGIN SOLUTION
<code>


Problem:
I have the following data frame:
import pandas as pd
import io
from scipy import stats
temp=u"""probegenes,sample1,sample2,sample3
1415777_at Pnliprp1,20,0.00,11
1415805_at Clps,17,0.00,55
1415884_at Cela3b,47,0.00,100"""
df = pd.read_csv(io.StringIO(temp),index_col='probegenes')
df
It looks like this
                     sample1  sample2  sample3
probegenes
1415777_at Pnliprp1       20        0       11
1415805_at Clps           17        0       55
1415884_at Cela3b         47        0      100
What I want to do is too perform column-zscore calculation using SCIPY. At the end of the day. the result will look like:
                               sample1  sample2  sample3
probegenes
1415777_at Pnliprp1             x.xxxxxxxx,    x.xxxxxxxx,  x.xxxxxxxx
1415805_at Clps                 x.xxxxxxxx,    x.xxxxxxxx,  x.xxxxxxxx
1415884_at Cela3b               x.xxxxxxxx,    x.xxxxxxxx,  x.xxxxxxxx
A:
<code>
import pandas as pd
import io
from scipy import stats

temp=u"""probegenes,sample1,sample2,sample3
1415777_at Pnliprp1,20,0.00,11
1415805_at Clps,17,0.00,55
1415884_at Cela3b,47,0.00,100"""
df = pd.read_csv(io.StringIO(temp),index_col='probegenes')
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>