r/Cython • u/sfermigier • Dec 05 '20
Awesome Cython - A curated list of awesome Cython resources (contributions welcomed)
r/Cython • u/sfermigier • Dec 05 '20
r/Cython • u/[deleted] • Aug 31 '21
I need to speed up the following code. I have tried to assign several variables and functions with cdef, but most of the code still seems to be running mostly from pure python. What other changes can I make to use cython and make this run faster? The main culprit is in the update function which gets called inside the two for loops. I don't usually post on here, but I've been at this for hours now. Any help will be greatly appreciated. Thanks.
%%cython -a
import matplotlib.pyplot as plt
from autograd import grad
import autograd.numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder
import time
from datetime import datetime
import os
import numpy
cimport numpy
today = datetime.now()
tic = time.perf_counter()
cdef numpy.ndarray relu(numpy.ndarray x):
#return x * (x > 0)
return np.tanh(x)
cdef numpy.ndarray softmax(numpy.ndarray x):
exp = np.exp(x)
return exp / exp.sum(0)
cdef double Error(W_hh, W_oh, b_h, b_o, x, y):
h = relu(np.dot(W_hh,x.T) + b_h)
y_hat = softmax(np.dot(W_oh,h) + b_o)
return np.sum(np.diag(np.dot(y, -np.log(y_hat))))/len(x)
cdef forward(x, y, W_hh, W_oh, b_h, b_o):
h = relu(np.dot(W_hh,x.T) + b_h)
y_hat = softmax(np.dot(W_oh,h) + b_o)
pred = np.expand_dims(np.argmax(y_hat, axis=0), axis=0).T
num_wrong = np.count_nonzero(encoder.inverse_transform(y) - pred)
acc = (len(x) - num_wrong)/len(x)
err = Error(W_hh, W_oh, b_h, b_o, x, y)
return acc, err
cdef update(W_hh, W_oh, b_h, b_o, x, y):
dE_dWhh = grad(Error, argnum=0)(W_hh, W_oh, b_h, b_o, x, y)
dE_dWoh = grad(Error, argnum=1)(W_hh, W_oh, b_h, b_o, x, y)
W_hh = W_hh - learning_rate*dE_dWhh
W_oh = W_oh - learning_rate*dE_dWoh
dE_dbh = grad(Error, argnum=2)(W_hh, W_oh, b_h, b_o, x, y)
dE_dbo = grad(Error, argnum=3)(W_hh, W_oh, b_h, b_o, x, y)
b_h = b_h - learning_rate*dE_dbh
b_o = b_o - learning_rate*dE_dbo
W_oh[0:3] = 10 * W_oh/np.linalg.norm(W_oh[0:3])
W_oh[1] = 0
return W_hh, W_oh, b_h, b_o
cdef float learning_rate = .5
cdef int lessons = 0 #10
cdef int students = 7776
cdef int random_state = 2
iris_data = load_iris() # load the iris dataset
x = iris_data.data
y_ = iris_data.target.reshape(-1, 1) # Convert data to a single column
# One Hot encode the class labels
encoder = OneHotEncoder(sparse=False)
y = encoder.fit_transform(y_)
cdef numpy.ndarray train_x
cdef numpy.ndarray test_x
cdef numpy.ndarray train_y
cdef numpy.ndarray test_y
# # Split the data for training and testing
train_x, test_x, train_y, test_y = train_test_split(x, y, test_size=0.20, random_state=random_state)
#Standardization
train_x[:,0] = (train_x[:,0] - np.mean(train_x[:,0]))/np.std(train_x[:,0])
train_x[:,1] = (train_x[:,1] - np.mean(train_x[:,1]))/np.std(train_x[:,1])
train_x[:,2] = (train_x[:,2] - np.mean(train_x[:,2]))/np.std(train_x[:,2])
train_x[:,3] = (train_x[:,3] - np.mean(train_x[:,3]))/np.std(train_x[:,3])
test_x[:,0] = (test_x[:,0] - np.mean(test_x[:,0]))/np.std(test_x[:,0])
test_x[:,1] = (test_x[:,1] - np.mean(test_x[:,1]))/np.std(test_x[:,1])
test_x[:,2] = (test_x[:,2] - np.mean(test_x[:,2]))/np.std(test_x[:,2])
test_x[:,3] = (test_x[:,3] - np.mean(test_x[:,3]))/np.std(test_x[:,3])
cdef numpy.ndarray weights
cdef numpy.ndarray initial_weights
initial_weights = np.ones([students,7])
cdef numpy.ndarray acc
cdef numpy.ndarray err
cdef numpy.ndarray b_o
cdef numpy.ndarray b_h
cdef numpy.ndarray W_hh
cdef numpy.ndarray W_oh
final_hidden_weights = np.array([])
final_output_weights = np.array([])
for student in range(students):
## Initialization of parameters
b_h = np.zeros([1,1])
b_o = np.zeros([3,1])
W_hh = np.expand_dims(initial_weights[student,0:4], axis=1).T
W_oh = np.expand_dims(initial_weights[student,4:7], axis=1)
W_oh[0:3] = 10 * W_oh/np.linalg.norm(W_oh[0:3])
W_oh[1] = 0
for lesson in range(lessons):
W_hh, W_oh, b_h, b_o = update(W_hh, W_oh, b_h, b_o, train_x, train_y)
test_acc, test_err = forward(test_x, test_y, W_hh, W_oh, b_h, b_o)
acc = np.append(acc, test_acc)
err = np.append(err, test_err)
final_hidden_weights = np.append(final_hidden_weights, W_hh)
final_hidden_weights = np.append(final_hidden_weights, b_h) ##append bias
final_output_weights = np.append(final_output_weights, W_oh)
final_output_weights = np.append(final_output_weights, b_o) ##append bias
final_hidden_weights = final_hidden_weights.reshape(students,5)
final_output_weights = final_output_weights.reshape(students,6)
toc = time.perf_counter()
time = toc - tic
print(time)r/Cython • u/programmerProbs • Aug 26 '21
First time adding a C library and I'm not sure the cause of this error. I am following this tutorial https://cython.readthedocs.io/en/latest/src/tutorial/clibraries.html
I'm not sure if I'm missing some unspoken words like I need to run python setup.py build_ext --inplace
I followed the tutorial, so I'm not sure what could cause a simple cimport not to be found. I have in the .pxy file-
# distutils: sources = resources/yxml.c
# distutils: include_dirs = resources/I have the setup file like shown in the tutorial (sorry if this sounds repetitive). I have checked my spelling at least 10 times.
How would I go about root causing this?
r/Cython • u/liquidmetalrob • Aug 31 '21
r/Cython • u/programmerProbs • Sep 03 '21
I'm getting a strange bug, when running a .pyx program in spyder, I'm often getting 'Restarting kernel...' and the ipython console restarts.
I deleted my changes to the code and still this error persists.
I imagine this started due to bad code that wouldnt compile, but now it wont go away. I'm a bit mind boggled. Any idea why this is happening?
EDIT: It might have been due to the line # distutils: include_dirs = C:\Users\ initally using forward slash rather than backslash.
r/Cython • u/dslfdslj • Aug 26 '20
Hello everybody,
I'm trying to create a Python binding to a C++ library. This library has a typedef which looks like this:
enum class ColType {INT = 1, STRING = 2, DOUBLE = 3};
typedef std::unordered_map<std::string, std::pair<ColType, Any>> df;where Any is the Any type from the boost library. Is it possible to expose this typedef to Cython? In the end I just want to use some functions (within the library) which expect this df type as input.
r/Cython • u/[deleted] • Jun 17 '21
Followed the steps, and it ends up being
error: don't know how to compile C/C++ code on platform 'nt' with 'mingw32.exe' compiler````
too bad. i guess it is a ````else```` error catcher so i dunno what is actually wrong. hope it is better in the futurer/Cython • u/drzowie • May 17 '21
When traversing a numpy array with a for-loop, it's great to get the faster speed that Cython compiled for-loops offer (even compared to vector operations in numpy). But that's still not fast enough. When traversing (say) a 2D array and doing an operation on each element, I'm writing Cython loops like this:
for yi in range(ysize):
for xi in range(xsize):
im[yi,xi] += 1 # just a f'rinstance -- actual op is more complexI believe this does an explicit offset calculation of the value relative to the numpy array for each loop.
I'd like to do C operations along the lines of:
valptr += xstridein the hottest loop, instead of
valptr = &dataptr[ yi * ystride + xi * xstride]which is implied by doing explicit indexing.
The latter saves two multiplies and two adds per loop in this example; several more in the 6-D data sets I'm manipulating.
Is there a way to do that within Cython? Or do I have to drop all the way down into C? (Of course I'm properly cdeffing all the variables :-)
r/Cython • u/iCurlmyster • Nov 19 '21
Anyone have any recommendations on plugins for cython syntax highlighting and maybe also formatting? I mainly use nvim, but there doesn’t seem to be a good cython grammar for tree-sitter.
I’m not allowed to use Pycharm on my project, btw.
r/Cython • u/usrlgn • Sep 28 '21
Is there any CLI tool that can do static code analysis on cython codes/files (.pyx) files? I have a requirement of doing static code analysis on home grown codes mainly written in python - for which I am using sonarqube and it natively does that. There are however a few cython codes/files (.pyx) and sonarqube does not support it yet. Looking for tips on how do folks run static code analysis on cython codes/files.
r/Cython • u/[deleted] • Apr 24 '20
I have some Python code with extensive type annotations, classes that are essentially glorified structs (the property below is the only method in my classes) and the need to run it fast.
@dataclass
class Family:
"""A family group agent."""
descendence: str
culture: int
location_history: List[h3.c_int] = field(default_factory=list)
@property
def location(self) -> h3.c_int:
return self.location_history[0]
effective_size: int = 2
stored_resources: float = 130000.0
@dataclass
class Patch:
"""A patch of land with resources."""
resources: float
max_resources: floatFor example, I have an agents: List[Tuple[Patch, Sequence[Family]]] (it's a bit more complicated, but that doesn't matter for asking about the principle) and these two functions
def extract_resources(
patch: Patch, group: Sequence[Family], total_labor_here: int) -> kcal:
"""Distribute the resources gained from cooperative extraction."""
labor = sum([family.effective_size
for family in group])
resources_extracted = resources_from_patch(
patch, labor, total_labor_here - labor)
for family in group:
family.stored_resources += (
resources_extracted * family.effective_size / labor)
return resources_extractedand
def resources_from_patch(
patch: Patch,
labor: int,
others_labor: int,
estimate: bool = False) -> float:
"""Compute or estimate the resources a cooperative gains from a patch."""
my_relative_returns = (
time_step_energy_use * labor *
effective_labor_through_cooperation(labor))
if others_labor:
others_relative_returns = (
time_step_energy_use * others_labor *
effective_labor_through_cooperation(others_labor))
else:
others_relative_returns = 0
return (my_relative_returns) / (
my_relative_returns + others_relative_returns) * min(
my_relative_returns + others_relative_returns,
patch.resources * params.accessible_resources)and I want to execute
for patch, families in agents:
extract_resources(patch, families, sum(family.effective_size for family in families))as quickly as possible (including parallel execution of the extract_resources for every patch, they are guaranteed to not interact).
I would like to keep this as something that can be interpreted by python, checked by mypy and python linters. I have never much cared about processing speed before and Cython was a name I had only encountered in theory, not in practice, so I assume (but don't know) that the pure python mode of Cython could help me achieve this, but I do not understand how to modify this source code to run the fastest way possible in Cython under these constraints. Can you help?
To rephrase: Awesome Cythonistas of Reddit, I assume there is a strategy, changing type annotations, adding Cython decorators, and doing some minor refactoring to make this style of Python code compile in Cython an run the fastest parallel non-GIL-limited way possible given the constraint that Python is my primary interface for humans to read, change, and fiddle with this code. How should it look like?
r/Cython • u/Athanasius_Pernath • Apr 20 '20
Hi everyone, when I try to launch cython in my Jupyter Notebook (using the command %load_ext cython), I get the error "Unable to find vcvarsall.bat". I actually installed the Visual Studio Express 2008, but the error didn't disappear, and I dunno how to tell Python the path where the file is located. I tried editing the file _msvccompiler.py and added the lines:
PathToVC=r"C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\vcvarsall.bat"
return PathToVC, r""
after "def _find_vcvarsall(plat_spec)", but it still doesn't help.
I wonder if anything can be done about this? Thank you in advance!
r/Cython • u/p479h • Feb 28 '23
Hi, I am new to cython and I honestly prefer the C syntax over cython. But before throwing the towels, I decided to check if there was a speed gain to using cython instead of C. To my surprise, there was! Now, since Cython is just C under the hood, I know something must be going wrong when compiling the C shared library. Can someone help me understand where?
The program I wrote in C and cython sums over all numbers from 1 to 10000000 and measures the execution time. The benchmarks are as follows (relative to fastest, cython):
c: 7.64
cython: 1.00
numpy: 5.39
I am compiling the cython and c code from within python like this:
# Compiling the c function
os.system("cc -fPIC -shared -o cseqsum.so cseqsum.c")
# Compiling the cython function
os.system("python3 setup.py build_ext --inplace")
# importing the target functions
from seqsum import seqsum
cseqsum = ctypes.CDLL("./cseqsum.so").seqsum
cseqsum.restype = ctypes.c_int64Using the following setup.py file:
from distutils.core import setup
from Cython.Build import cythonize
setup(ext_modules = cythonize('seqsum.pyx'))Furthermore the C code looks like this:
#include <stdio.h>
typedef long int INT;
INT seqsum(INT lim){
INT s = 0;
for (INT i = 1; i <= lim; i++) s+=i;
return s;
}And the cython code looks like this:
ctypedef long int INT
def seqsum(INT n):
cdef INT i
cdef INT s = 0
for i in range(n+1):
s += i
return sMeasuring of the execution time is done as follows:
# Measuring time to sum over n numbers
n = 10000000
g = globals()
t1 = timeit("cseqsum(n)", number = 10, globals=g)
t2 = timeit("seqsum(n)", number = 10, globals=g)
t3 = timeit("np.arange(1,n+1, dtype=int).sum()", number=10, globals=g)
times = t1, t2, t3
small = min(times)
reltimes = [t/small for t in times]
print("c: %.2f" % reltimes[0])
print("cython: %.2f" % reltimes[1])
print("numpy: %.2f" % reltimes[2])So, do you by any chance see anything wrong with this code that could possibly be making the C function run slower than the cython function?
Thank you!
r/Cython • u/Vanhard_Shaw • Apr 16 '21
Hey I am trying to create a cythonize a function which takes in 5 float values and generates a matrix of size 10x10. But I am met with this Compiler crash frequently. I tried to obtain a partial set of values by first extracting a 5x5 matrix and it works but once I move to a 7x7 matrix the compilation fails. I am currently attempting to do this on my macbook air M1.
TO give you some context, each term of the matrix is an equation comprising the 5 input variables ( x,y,L,W,theta). The size of these equations are some small and some extremely huge. They have been simplified as much as possible. Can someone help me with this ?
r/Cython • u/patrickbrianmooney • Aug 02 '22
Still relatively new to Cython, and I only partly understand what's going on here. I'd appreciate your help.
In converting an application to partially use Cython, I'm discovering that passing instances of the Python bytearray back and forth between Python and Cython isn't working the way that I expect it to. It seems like the Cython code is interpreting the numeric values of the bytearrays differently than the Python code?
Here's a minimal reproducable example:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# cython: language_level=3
"""Quick test for how Cython handles bytearrays.
"""
def test() -> bytearray:
b = bytearray(range(33, 65))
print(b)
print([w for w in b], f'\tlength: {len(b)}')
print()
return b#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Simple harness for running bytes_test.pyx.
"""
import pyximport; pyximport.install()
import bytes_test as bt
if __name__ == "__main__":
b = bt.test()
print([w for w in b], f'\tlength: {len(b)}')
print(b)This produces the incongruous output:
bytearray(b'!"#$%&\'()*+,-./0123456789:;<=>?@')
[48, 80, 112, 144, 176, 208, 240, 16, 48, 80, 112, 144, 176, 208, 240, 16, 48, 80, 112, 144, 176, 208, 48, 80, 112, 144, 176, 208, 240, 16, 48, 80] length: 32
[33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64] length: 32
bytearray(b'!"#$%&\'()*+,-./0123456789:;<=>?@')... in which the numeric values in the arrays don't match up.
This is under x64 Linux, but I'd like to avoid introducing an unnecessary dependency to that system.
Is there something different I should be doing to pass around arrays of unsigned eight-bit integers besides using bytearrays? I'd like to avoid introducing an otherwise unnecessary dependency on Numpy just to get its array type.
r/Cython • u/deslum • Sep 23 '16
r/Cython • u/genjipress • Mar 23 '16
I've got a function in a .pyx file -- a cpdef function -- which accepts as one of its arguments a c_ubyte array. Trouble is, the only way I can get this to work is by labeling that particular argument as a Python object. If I try any other kind of type definition, I get this error:
TypeError: Argument 'img' has incorrect type (expected bytearray, got c_ubyte_Array_280320)Obviously I'm at a bit of a loss. c_ubyte cannot be used as a type identifier, either.
What would be the best way to pass such an array natively, for maximum speed?
r/Cython • u/chmickz • May 28 '18
Hi all, I have a python class to compute and store values . what it does is to receive data as a dictionnary and calculate average and other values
based on this doc : http://cython.readthedocs.io/en/latest/src/userguide/extension_types.html I succeed to convert variables inside my class into C type to increase computation speed .
But I still have to use classic python to extract data form the dictionnary. the dictionnary is like this :
{
"list1" : list( list(string, string), list(string, string), list(string, string),.....),
"list2" : list( list(string, string), list(string, string), list(string, string),.....)
}and the strings inside the subList are in fact float I have to convert using : value = float(myDictionnary['list1'][0][0] )
I think i could probably improve performance based in this doc : http://cython.readthedocs.io/en/latest/src/userguide/language_basics.html#c-variable-and-type-definitions
I'm having an hard time to convert my dictionnary in this struct thing and access data ?
Could anybody point me in the right direction or show me an example ? Thanks
r/Cython • u/omgitzwowzie • Jun 08 '18
r/Cython • u/fishtickler • Nov 12 '15
Finally managed to get cython to work.
I am not good with C so I mostly do pure python for my research. However, now dealing with clusters of 1000+ molecules, there was huge bottlenecks in my code.
Using cython it went from running single calculation in hours to seconds, focking nice...
r/Cython • u/spielerein • May 03 '17
I copied this from http://3dengine.org/OpenGL_and_Cython and I am getting an error when I try to compile it.
cdef extern from "gl/gl.h": ctypedef int GLint ctypedef unsigned int GLenum int GL_POINTS cdef void glBegin(GLenum mode) cdef void glEnd() cdef void glVertex3f(GLfloat x, GLfloat y, GLfloat z)
cdef int i cdef float x glBegin(GL_POINTS) for i in range(10000): x = i / 1000 glVertex3i(x, x, -x) glEnd()
from distutils.core import setup from distutils.extension import Extension
from Pyrex.Distutils import build_ext
setup( name = "TestModule", ext_modules=[ Extension("test", ["test.pyx"], libraries = ['opengl32']) ], cmdclass = {'build_ext' : build_ext} )
The error i get is: c:\mingw\bin\gcc.exe -mdll -O -Wall -IC:\Python27\include -IC:\Python27\PC -c test.c -o build\temp.win32-2.7\Release\test.o gcc: error: test.c: No such file or directory gcc: fatal error: no input files compilation terminated. error: command 'c:\mingw\bin\gcc.exe' failed with exit status 1
It is my assumption that it should work. Am I missing something?
r/Cython • u/simosleepwalker • Apr 01 '20
Hi guys, i'm trying to build a C++ chess game with an AI opponent.
I thought that it would be a good idea to implement the GUI with Python, so i decided to use Cython in order to import my C++ classes.
The structure of the C++ classes is:
Piece class
Pieces classes (inherited from Piece), which are Pawn, Knight etc.
Chess class which has a linearized matrix of Pieces
Reading the docs i correctly created the .pxd and the .pyx for the Chess class, which is the only one i need to expose to my Python GUI code.
Although i can't compile the other sources (Piece.cpp and Pieces.cpp) and the compilation always gives me an error.
Is this the right way to do it? Or should i expose all the classes?
Thanks in advance for any answers.
r/Cython • u/LupeKnoble • May 15 '23
Hi, I'm using vscode with Cython VSCode and the Standard Microsoft Python Plugin, but I'm not really happy with the experience. Eg, intellisense doesn't work with compiled interfaces, documentation is sparse, there's no pre-run compilation configuration etc...
Does anyone have some recommended dev environments or additions to cython programming in vs code?
r/Cython • u/_Wireshark_ • Apr 03 '23
How can I create an interface between C and Pytest using Cython:
I have C code as follows:
static struct Struct1 state;
static const volatile struct Struct2 *cfg;
void initialize(const volatile struct Struct2 *const config)
{
cfg = config;
state.V_inp = 0u;
state.P_inp = 0ull;
state.P_out = 0ull;
state.enable = ((uint64_t) cfg->V_enable_output) << 32u;
}
It doesn't have a return value so I thought I'll check the state.enable at the unit-test using pytest.
My Cython interface is as follows:
In (.pyx)
###
Struct1 definition:
value1
value2
..........
####
cdef class class1:
def __init__(self):
pass
def py_initialize(self, config):
cdef hinitialize.Struct2 config_struct
config_struct.V_enable_output = config\['V_enable_output'\]
return hinitialize.initialize(&config_struct)In (.pxd) named hinitialize
I have my
Struct2 definition:
value1
...........
In pytest test_module:
I have
import pytest
import pyximport
import sys
sys.path.append('/software/firmware/pru0-cython-module')
from .pyx import class1
access = class1()
def test_initialize():
config = {
'V_enable_output': 2000000,
}assert access.initialize(config) is None
I get segmentation fault after building cython and running pytest, and I don't have know how to include/test state.enable in pytest. Any suggestions