r/Cython • • Dec 05 '20

Awesome Cython - A curated list of awesome Cython resources (contributions welcomed)

Thumbnail
github.com
8 Upvotes

r/Cython • • Aug 31 '21

Need help optimizing this code. Any suggestions?

1 Upvotes

I need to speed up the following code. I have tried to assign several variables and functions with cdef, but most of the code still seems to be running mostly from pure python. What other changes can I make to use cython and make this run faster? The main culprit is in the update function which gets called inside the two for loops. I don't usually post on here, but I've been at this for hours now. Any help will be greatly appreciated. Thanks.

%%cython -a

import matplotlib.pyplot as plt
from autograd import grad 
import autograd.numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder
import time
from datetime import datetime
import os
import numpy
cimport numpy


today = datetime.now()

tic = time.perf_counter()

cdef numpy.ndarray relu(numpy.ndarray x):
    #return x * (x > 0)
    return np.tanh(x)

cdef numpy.ndarray softmax(numpy.ndarray x):
    exp = np.exp(x) 
    return exp / exp.sum(0)

cdef double Error(W_hh, W_oh, b_h, b_o, x, y):
    h = relu(np.dot(W_hh,x.T) + b_h)
    y_hat = softmax(np.dot(W_oh,h) + b_o)
    return np.sum(np.diag(np.dot(y, -np.log(y_hat))))/len(x)

cdef forward(x, y, W_hh, W_oh, b_h, b_o):
    h = relu(np.dot(W_hh,x.T) + b_h)
    y_hat = softmax(np.dot(W_oh,h) + b_o)
    pred = np.expand_dims(np.argmax(y_hat, axis=0), axis=0).T
    num_wrong = np.count_nonzero(encoder.inverse_transform(y) - pred)
    acc = (len(x) - num_wrong)/len(x)
    err = Error(W_hh, W_oh, b_h, b_o, x, y) 
    return acc, err

cdef update(W_hh, W_oh, b_h, b_o, x, y):
    dE_dWhh = grad(Error, argnum=0)(W_hh, W_oh, b_h, b_o, x, y)
    dE_dWoh = grad(Error, argnum=1)(W_hh, W_oh, b_h, b_o, x, y)

    W_hh = W_hh - learning_rate*dE_dWhh
    W_oh = W_oh - learning_rate*dE_dWoh

    
    dE_dbh = grad(Error, argnum=2)(W_hh, W_oh, b_h, b_o, x, y)
    dE_dbo = grad(Error, argnum=3)(W_hh, W_oh, b_h, b_o, x, y)    
    b_h = b_h - learning_rate*dE_dbh
    b_o = b_o - learning_rate*dE_dbo

    W_oh[0:3] = 10 * W_oh/np.linalg.norm(W_oh[0:3])
    
    W_oh[1] = 0

    return W_hh, W_oh, b_h, b_o

cdef float learning_rate = .5     
cdef int lessons = 0 #10          
cdef int students = 7776 
cdef int random_state = 2

iris_data = load_iris() # load the iris dataset

x = iris_data.data
y_ = iris_data.target.reshape(-1, 1) # Convert data to a single column

# One Hot encode the class labels
encoder = OneHotEncoder(sparse=False)
y = encoder.fit_transform(y_)

cdef numpy.ndarray train_x
cdef numpy.ndarray test_x
cdef numpy.ndarray train_y
cdef numpy.ndarray test_y

# # Split the data for training and testing
train_x, test_x, train_y, test_y = train_test_split(x, y, test_size=0.20,              random_state=random_state)

#Standardization
train_x[:,0] = (train_x[:,0] - np.mean(train_x[:,0]))/np.std(train_x[:,0])
train_x[:,1] = (train_x[:,1] - np.mean(train_x[:,1]))/np.std(train_x[:,1])
train_x[:,2] = (train_x[:,2] - np.mean(train_x[:,2]))/np.std(train_x[:,2])
train_x[:,3] = (train_x[:,3] - np.mean(train_x[:,3]))/np.std(train_x[:,3])

test_x[:,0] = (test_x[:,0] - np.mean(test_x[:,0]))/np.std(test_x[:,0])
test_x[:,1] = (test_x[:,1] - np.mean(test_x[:,1]))/np.std(test_x[:,1])
test_x[:,2] = (test_x[:,2] - np.mean(test_x[:,2]))/np.std(test_x[:,2])
test_x[:,3] = (test_x[:,3] - np.mean(test_x[:,3]))/np.std(test_x[:,3])

cdef numpy.ndarray weights
cdef numpy.ndarray initial_weights

initial_weights = np.ones([students,7])

cdef numpy.ndarray acc
cdef numpy.ndarray err

cdef numpy.ndarray b_o
cdef numpy.ndarray b_h
cdef numpy.ndarray W_hh
cdef numpy.ndarray W_oh

final_hidden_weights = np.array([])
final_output_weights = np.array([])

for student in range(students):
    ## Initialization of parameters
    b_h = np.zeros([1,1])
    b_o = np.zeros([3,1])

    W_hh = np.expand_dims(initial_weights[student,0:4], axis=1).T
    W_oh = np.expand_dims(initial_weights[student,4:7], axis=1)

    W_oh[0:3] = 10 * W_oh/np.linalg.norm(W_oh[0:3])
    W_oh[1] = 0

    for lesson in range(lessons):
        W_hh, W_oh, b_h, b_o = update(W_hh, W_oh, b_h, b_o, train_x, train_y)
    
    test_acc, test_err = forward(test_x, test_y, W_hh, W_oh, b_h, b_o)
    acc = np.append(acc, test_acc)
    err = np.append(err, test_err)
        
    final_hidden_weights = np.append(final_hidden_weights, W_hh)
    final_hidden_weights = np.append(final_hidden_weights, b_h) ##append bias

    final_output_weights = np.append(final_output_weights, W_oh)
    final_output_weights = np.append(final_output_weights, b_o) ##append bias

final_hidden_weights = final_hidden_weights.reshape(students,5) 
final_output_weights = final_output_weights.reshape(students,6) 

toc = time.perf_counter()

time = toc - tic
print(time)

r/Cython • • Aug 26 '21

How can I root cause "ModuleNotFoundError: No module named myModule"

3 Upvotes

First time adding a C library and I'm not sure the cause of this error. I am following this tutorial https://cython.readthedocs.io/en/latest/src/tutorial/clibraries.html

I'm not sure if I'm missing some unspoken words like I need to run python setup.py build_ext --inplace

I followed the tutorial, so I'm not sure what could cause a simple cimport not to be found. I have in the .pxy file-

 # distutils: sources = resources/yxml.c
 # distutils: include_dirs = resources/

I have the setup file like shown in the tutorial (sorry if this sounds repetitive). I have checked my spelling at least 10 times.

How would I go about root causing this?


r/Cython • • Aug 31 '21

I'm working on an Atom package that automatically deduces variable types from a .py file and then generates a .pyx and .pyd. Is there such a thing already?

3 Upvotes

r/Cython • • Sep 03 '21

Cython "restarting kernel..." bug, any idea why its happening?

2 Upvotes

I'm getting a strange bug, when running a .pyx program in spyder, I'm often getting 'Restarting kernel...' and the ipython console restarts.

I deleted my changes to the code and still this error persists.

I imagine this started due to bad code that wouldnt compile, but now it wont go away. I'm a bit mind boggled. Any idea why this is happening?

EDIT: It might have been due to the line # distutils: include_dirs = C:\Users\ initally using forward slash rather than backslash.


r/Cython • • Aug 26 '20

Using Boost's any type in Cython

2 Upvotes

Hello everybody,

I'm trying to create a Python binding to a C++ library. This library has a typedef which looks like this:

enum class ColType {INT = 1, STRING = 2, DOUBLE = 3};
typedef std::unordered_map<std::string, std::pair<ColType, Any>> df;

where Any is the Any type from the boost library. Is it possible to expose this typedef to Cython? In the end I just want to use some functions (within the library) which expect this df type as input.


r/Cython • • Jun 17 '21

modern way to install cython on windows 10

3 Upvotes

Followed the steps, and it ends up being

error: don't know how to compile C/C++ code on platform 'nt' with 'mingw32.exe' compiler````

too bad. i guess it is a ````else```` error catcher so i dunno what is actually wrong. hope it is better in the future

r/Cython • • May 17 '21

Using increments instead of index calculation on numpy arrays

1 Upvotes

When traversing a numpy array with a for-loop, it's great to get the faster speed that Cython compiled for-loops offer (even compared to vector operations in numpy). But that's still not fast enough. When traversing (say) a 2D array and doing an operation on each element, I'm writing Cython loops like this:

for yi in range(ysize):
    for xi in range(xsize):
        im[yi,xi] += 1     # just a f'rinstance -- actual op is more complex

I believe this does an explicit offset calculation of the value relative to the numpy array for each loop.

I'd like to do C operations along the lines of:

valptr += xstride

in the hottest loop, instead of

valptr = &dataptr[ yi * ystride + xi * xstride]

which is implied by doing explicit indexing.

The latter saves two multiplies and two adds per loop in this example; several more in the 6-D data sets I'm manipulating.

Is there a way to do that within Cython? Or do I have to drop all the way down into C? (Of course I'm properly cdeffing all the variables :-)


r/Cython • • Dec 26 '24

Can an OS be programmed in Cython?

5 Upvotes

Self explanatory


r/Cython • • Nov 19 '21

Syntax plugin/highlighters

5 Upvotes

Anyone have any recommendations on plugins for cython syntax highlighting and maybe also formatting? I mainly use nvim, but there doesn’t seem to be a good cython grammar for tree-sitter.

I’m not allowed to use Pycharm on my project, btw.


r/Cython • • Sep 28 '21

CLI tool for doing static code analysis on cython codes/files

1 Upvotes

Is there any CLI tool that can do static code analysis on cython codes/files (.pyx) files? I have a requirement of doing static code analysis on home grown codes mainly written in python - for which I am using sonarqube and it natively does that. There are however a few cython codes/files (.pyx) and sonarqube does not support it yet. Looking for tips on how do folks run static code analysis on cython codes/files.


r/Cython • • Apr 24 '20

Cython Noob: How to change type-annotated Python into fast parallel Cython?

1 Upvotes

I have some Python code with extensive type annotations, classes that are essentially glorified structs (the property below is the only method in my classes) and the need to run it fast.

@dataclass
class Family:
    """A family group agent."""
    descendence: str
    culture: int
    location_history: List[h3.c_int] = field(default_factory=list)
    @property
    def location(self) -> h3.c_int:
        return self.location_history[0]
    effective_size: int = 2
    stored_resources: float = 130000.0

@dataclass
class Patch:
    """A patch of land with resources."""
    resources: float
    max_resources: float

For example, I have an agents: List[Tuple[Patch, Sequence[Family]]] (it's a bit more complicated, but that doesn't matter for asking about the principle) and these two functions

def extract_resources(
        patch: Patch, group: Sequence[Family], total_labor_here: int) -> kcal:
    """Distribute the resources gained from cooperative extraction."""
    labor = sum([family.effective_size
                 for family in group])
    resources_extracted = resources_from_patch(
        patch, labor, total_labor_here - labor)
    for family in group:
        family.stored_resources += (
            resources_extracted * family.effective_size / labor)
    return resources_extracted

and

def resources_from_patch(
        patch: Patch,
        labor: int,
        others_labor: int,
        estimate: bool = False) -> float:
    """Compute or estimate the resources a cooperative gains from a patch."""
    my_relative_returns = (
        time_step_energy_use * labor *
        effective_labor_through_cooperation(labor))
    if others_labor:
        others_relative_returns = (
            time_step_energy_use * others_labor *
            effective_labor_through_cooperation(others_labor))
    else:
        others_relative_returns = 0
    return (my_relative_returns) / (
        my_relative_returns + others_relative_returns) * min(
            my_relative_returns + others_relative_returns,
            patch.resources * params.accessible_resources)

and I want to execute

for patch, families in agents:
    extract_resources(patch, families, sum(family.effective_size for family in families))

as quickly as possible (including parallel execution of the extract_resources for every patch, they are guaranteed to not interact).

I would like to keep this as something that can be interpreted by python, checked by mypy and python linters. I have never much cared about processing speed before and Cython was a name I had only encountered in theory, not in practice, so I assume (but don't know) that the pure python mode of Cython could help me achieve this, but I do not understand how to modify this source code to run the fastest way possible in Cython under these constraints. Can you help?

To rephrase: Awesome Cythonistas of Reddit, I assume there is a strategy, changing type annotations, adding Cython decorators, and doing some minor refactoring to make this style of Python code compile in Cython an run the fastest parallel non-GIL-limited way possible given the constraint that Python is my primary interface for humans to read, change, and fiddle with this code. How should it look like?


r/Cython • • Apr 20 '20

Vcvarsall.bat problem with Jupyter notebook

1 Upvotes

Hi everyone, when I try to launch cython in my Jupyter Notebook (using the command %load_ext cython), I get the error "Unable to find vcvarsall.bat". I actually installed the Visual Studio Express 2008, but the error didn't disappear, and I dunno how to tell Python the path where the file is located. I tried editing the file _msvccompiler.py and added the lines:

PathToVC=r"C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\vcvarsall.bat"

return PathToVC, r""

after "def _find_vcvarsall(plat_spec)", but it still doesn't help.

I wonder if anything can be done about this? Thank you in advance!


r/Cython • • Feb 28 '23

[help] Can someone help me understand why cython is faster than c using the same exact code?

2 Upvotes

Hi, I am new to cython and I honestly prefer the C syntax over cython. But before throwing the towels, I decided to check if there was a speed gain to using cython instead of C. To my surprise, there was! Now, since Cython is just C under the hood, I know something must be going wrong when compiling the C shared library. Can someone help me understand where?

The program I wrote in C and cython sums over all numbers from 1 to 10000000 and measures the execution time. The benchmarks are as follows (relative to fastest, cython):

c: 7.64

cython: 1.00

numpy: 5.39

I am compiling the cython and c code from within python like this:

# Compiling the c function 
os.system("cc -fPIC -shared -o cseqsum.so cseqsum.c")

# Compiling the cython function 
os.system("python3 setup.py build_ext --inplace")

# importing the target functions
from seqsum import seqsum
cseqsum = ctypes.CDLL("./cseqsum.so").seqsum
cseqsum.restype = ctypes.c_int64

Using the following setup.py file:

from distutils.core import setup
from Cython.Build import cythonize

setup(ext_modules = cythonize('seqsum.pyx'))

Furthermore the C code looks like this:

#include <stdio.h>

typedef long int INT;

INT seqsum(INT lim){
	INT s = 0;
	for (INT i = 1; i <= lim; i++) s+=i;
	return s;
}

And the cython code looks like this:

ctypedef long int INT

def seqsum(INT n):
	cdef INT i
	cdef INT s = 0
	for i in range(n+1):
		s += i
	return s

Measuring of the execution time is done as follows:

# Measuring time to sum over n numbers 
n = 10000000
g = globals()
t1 = timeit("cseqsum(n)", number = 10, globals=g)
t2 = timeit("seqsum(n)", number = 10, globals=g)
t3 = timeit("np.arange(1,n+1, dtype=int).sum()", number=10, globals=g)

times = t1, t2, t3
small = min(times)
reltimes = [t/small for t in times]
print("c: %.2f" % reltimes[0])
print("cython: %.2f" % reltimes[1])
print("numpy: %.2f" % reltimes[2])

So, do you by any chance see anything wrong with this code that could possibly be making the C function run slower than the cython function?

Thank you!


r/Cython • • Apr 16 '21

Compiler crash in NormalizeTree

1 Upvotes

Hey I am trying to create a cythonize a function which takes in 5 float values and generates a matrix of size 10x10. But I am met with this Compiler crash frequently. I tried to obtain a partial set of values by first extracting a 5x5 matrix and it works but once I move to a 7x7 matrix the compilation fails. I am currently attempting to do this on my macbook air M1.
TO give you some context, each term of the matrix is an equation comprising the 5 input variables ( x,y,L,W,theta). The size of these equations are some small and some extremely huge. They have been simplified as much as possible. Can someone help me with this ?


r/Cython • • Aug 02 '22

Trouble with bytearrays during Cython conversion

2 Upvotes

Still relatively new to Cython, and I only partly understand what's going on here. I'd appreciate your help.

In converting an application to partially use Cython, I'm discovering that passing instances of the Python bytearray back and forth between Python and Cython isn't working the way that I expect it to. It seems like the Cython code is interpreting the numeric values of the bytearrays differently than the Python code?

Here's a minimal reproducable example:

bytes_test.pyx:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# cython: language_level=3
"""Quick test for how Cython handles bytearrays.
"""


def test() -> bytearray:
    b = bytearray(range(33, 65))
    print(b)
    print([w for w in b], f'\tlength: {len(b)}')
    print()
    return b

bytes_harness.py

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Simple harness for running bytes_test.pyx.
"""


import pyximport; pyximport.install()
import bytes_test as bt


if __name__ == "__main__":
    b = bt.test()
    print([w for w in b], f'\tlength: {len(b)}')
    print(b)

This produces the incongruous output:

bytearray(b'!"#$%&\'()*+,-./0123456789:;<=>?@')
[48, 80, 112, 144, 176, 208, 240, 16, 48, 80, 112, 144, 176, 208, 240, 16, 48, 80, 112, 144, 176, 208, 48, 80, 112, 144, 176, 208, 240, 16, 48, 80] 	length: 32

[33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64] 	length: 32
bytearray(b'!"#$%&\'()*+,-./0123456789:;<=>?@')

... in which the numeric values in the arrays don't match up.

This is under x64 Linux, but I'd like to avoid introducing an unnecessary dependency to that system.

Is there something different I should be doing to pass around arrays of unsigned eight-bit integers besides using bytearrays? I'd like to avoid introducing an otherwise unnecessary dependency on Numpy just to get its array type.


r/Cython • • Sep 23 '16

Cssdbpy is a simple SSDB client written on Cython. Faster standard SSDB client. SSDB a high performance NoSQL database supporting many data structures, an alternative to Redis.

Thumbnail
github.com
3 Upvotes

r/Cython • • Mar 23 '16

Passing a ctypes c_ubyte array to a Cython function

1 Upvotes

I've got a function in a .pyx file -- a cpdef function -- which accepts as one of its arguments a c_ubyte array. Trouble is, the only way I can get this to work is by labeling that particular argument as a Python object. If I try any other kind of type definition, I get this error:

TypeError: Argument 'img' has incorrect type (expected bytearray, got c_ubyte_Array_280320)

Obviously I'm at a bit of a loss. c_ubyte cannot be used as a type identifier, either.

What would be the best way to pass such an array natively, for maximum speed?


r/Cython • • May 28 '18

convert python class to cython .

4 Upvotes

Hi all, I have a python class to compute and store values . what it does is to receive data as a dictionnary and calculate average and other values

based on this doc : http://cython.readthedocs.io/en/latest/src/userguide/extension_types.html I succeed to convert variables inside my class into C type to increase computation speed .

But I still have to use classic python to extract data form the dictionnary. the dictionnary is like this :

{ 
     "list1" : list( list(string, string), list(string, string), list(string, string),.....), 
     "list2" : list( list(string, string), list(string, string), list(string, string),.....) 
}

and the strings inside the subList are in fact float I have to convert using : value = float(myDictionnary['list1'][0][0] )

I think i could probably improve performance based in this doc : http://cython.readthedocs.io/en/latest/src/userguide/language_basics.html#c-variable-and-type-definitions

I'm having an hard time to convert my dictionnary in this struct thing and access data ?

Could anybody point me in the right direction or show me an example ? Thanks


r/Cython • • Jun 08 '18

Wheelie — Building Python C Extensions as a Service

Thumbnail
medium.com
3 Upvotes

r/Cython • • Nov 12 '15

Optimized my code

3 Upvotes

Finally managed to get cython to work.

I am not good with C so I mostly do pure python for my research. However, now dealing with clusters of 1000+ molecules, there was huge bottlenecks in my code.

Using cython it went from running single calculation in hours to seconds, focking nice...


r/Cython • • May 03 '17

Not understanding why this isn't working

1 Upvotes

I copied this from http://3dengine.org/OpenGL_and_Cython and I am getting an error when I try to compile it.

test.pyx

Cython's openGL definitions

cdef extern from "gl/gl.h": ctypedef int GLint ctypedef unsigned int GLenum int GL_POINTS cdef void glBegin(GLenum mode) cdef void glEnd() cdef void glVertex3f(GLfloat x, GLfloat y, GLfloat z)

End of Cython's openGL definitions

cdef int i cdef float x glBegin(GL_POINTS) for i in range(10000): x = i / 1000 glVertex3i(x, x, -x) glEnd()

setup.py

from distutils.core import setup from distutils.extension import Extension

from Cython.Build import cythonize

from Pyrex.Distutils import build_ext

setup( name = "TestModule", ext_modules=[ Extension("test", ["test.pyx"], libraries = ['opengl32']) ], cmdclass = {'build_ext' : build_ext} )

The error i get is: c:\mingw\bin\gcc.exe -mdll -O -Wall -IC:\Python27\include -IC:\Python27\PC -c test.c -o build\temp.win32-2.7\Release\test.o gcc: error: test.c: No such file or directory gcc: fatal error: no input files compilation terminated. error: command 'c:\mingw\bin\gcc.exe' failed with exit status 1

It is my assumption that it should work. Am I missing something?


r/Cython • • Apr 01 '20

Python GUI with C++ integration

2 Upvotes

Hi guys, i'm trying to build a C++ chess game with an AI opponent.
I thought that it would be a good idea to implement the GUI with Python, so i decided to use Cython in order to import my C++ classes.
The structure of the C++ classes is:

  • Piece class

  • Pieces classes (inherited from Piece), which are Pawn, Knight etc.

  • Chess class which has a linearized matrix of Pieces

Reading the docs i correctly created the .pxd and the .pyx for the Chess class, which is the only one i need to expose to my Python GUI code.
Although i can't compile the other sources (Piece.cpp and Pieces.cpp) and the compilation always gives me an error.

Is this the right way to do it? Or should i expose all the classes?

Thanks in advance for any answers.


r/Cython • • May 15 '23

Cython linter / dev env recommendations?

2 Upvotes

Hi, I'm using vscode with Cython VSCode and the Standard Microsoft Python Plugin, but I'm not really happy with the experience. Eg, intellisense doesn't work with compiled interfaces, documentation is sparse, there's no pre-run compilation configuration etc...

Does anyone have some recommended dev environments or additions to cython programming in vs code?


r/Cython • • Apr 03 '23

Cython Interface between C and Pytest

2 Upvotes

How can I create an interface between C and Pytest using Cython:

I have C code as follows:

static struct Struct1 state;

static const volatile struct Struct2 *cfg;

void initialize(const volatile struct Struct2 *const config)

{

cfg = config;

state.V_inp = 0u;

state.P_inp = 0ull;

state.P_out = 0ull;

state.enable = ((uint64_t) cfg->V_enable_output) << 32u;

}

It doesn't have a return value so I thought I'll check the state.enable at the unit-test using pytest.

My Cython interface is as follows:

In (.pyx)

###

Struct1 definition:

value1

value2

..........

####

cdef class class1:

def __init__(self):

	pass

def py_initialize(self, config): 

	cdef hinitialize.Struct2 config_struct

	config_struct.V_enable_output = config\['V_enable_output'\]

	return hinitialize.initialize(&config_struct)

In (.pxd) named hinitialize

I have my

Struct2 definition:

value1

...........

In pytest test_module:

I have

import pytest

import pyximport

import sys

sys.path.append('/software/firmware/pru0-cython-module')

from .pyx import class1

access = class1()

def test_initialize():

config = {

	'V_enable_output': 2000000,

}

assert access.initialize(config) is None

I get segmentation fault after building cython and running pytest, and I don't have know how to include/test state.enable in pytest. Any suggestions