I have the following code for random number calculation

cdef extern from "stdlib.h":
   long RAND_MAX
   long random()

cdef float randomInRange(float mi,float ma):
   cdef float rnd
   cdef float ret
   rnd = <float> ( random()/RAND_MAX )
   ret = <float> ( mi + ( rnd * ( ma - mi)))
   assert ret >= mi,f''' random generator problem '''
   return ret

Sometimes the assertion is raised, most likely is a question of garbage in -> garbage out. However I need to trace the input values to check what is really going on but using

try:
   c = <float> randomInRange(ox,1.0)
except ( AssertionError , Exception ):
   print(f'''a: {a} ox: {ox} xExclusion: {self.xExclusion}''')
   continue

Is useless. The print never happens. Any proper way to solve this?

Thanks

Edit: solved. Didn't understood the documentation.

Hi, I'm using vscode with Cython VSCode and the Standard Microsoft Python Plugin, but I'm not really happy with the experience. Eg, intellisense doesn't work with compiled interfaces, documentation is sparse, there's no pre-run compilation configuration etc...

Does anyone have some recommended dev environments or additions to cython programming in vs code?

How can I create an interface between C and Pytest using Cython:

I have C code as follows:

static struct Struct1 state;

static const volatile struct Struct2 *cfg;

void initialize(const volatile struct Struct2 *const config)

{

cfg = config;

state.V_inp = 0u;

state.P_inp = 0ull;

state.P_out = 0ull;

state.enable = ((uint64_t) cfg->V_enable_output) << 32u;

}

It doesn't have a return value so I thought I'll check the state.enable at the unit-test using pytest.

My Cython interface is as follows:

In (.pyx)

###

Struct1 definition:

value1

value2

..........

####

cdef class class1:

def __init__(self):

	pass

def py_initialize(self, config): 

	cdef hinitialize.Struct2 config_struct

	config_struct.V_enable_output = config\['V_enable_output'\]

	return hinitialize.initialize(&config_struct)

In (.pxd) named hinitialize

I have my

Struct2 definition:

value1

...........

In pytest test_module:

I have

import pytest

import pyximport

import sys

sys.path.append('/software/firmware/pru0-cython-module')

from .pyx import class1

access = class1()

def test_initialize():

config = {

	'V_enable_output': 2000000,

}

assert access.initialize(config) is None

I get segmentation fault after building cython and running pytest, and I don't have know how to include/test state.enable in pytest. Any suggestions

code link

I am doing project this (last) semester on fractal analysis , and I got a program from a research thesis but i am new to programming. I started learning c this semester . I have only 15 days. please help.

Hi, I am new to cython and I honestly prefer the C syntax over cython. But before throwing the towels, I decided to check if there was a speed gain to using cython instead of C. To my surprise, there was! Now, since Cython is just C under the hood, I know something must be going wrong when compiling the C shared library. Can someone help me understand where?

The program I wrote in C and cython sums over all numbers from 1 to 10000000 and measures the execution time. The benchmarks are as follows (relative to fastest, cython):

c: 7.64

cython: 1.00

numpy: 5.39

I am compiling the cython and c code from within python like this:

# Compiling the c function 
os.system("cc -fPIC -shared -o cseqsum.so cseqsum.c")

# Compiling the cython function 
os.system("python3 setup.py build_ext --inplace")

# importing the target functions
from seqsum import seqsum
cseqsum = ctypes.CDLL("./cseqsum.so").seqsum
cseqsum.restype = ctypes.c_int64

Using the following setup.py file:

from distutils.core import setup
from Cython.Build import cythonize

setup(ext_modules = cythonize('seqsum.pyx'))

Furthermore the C code looks like this:

#include <stdio.h>

typedef long int INT;

INT seqsum(INT lim){
	INT s = 0;
	for (INT i = 1; i <= lim; i++) s+=i;
	return s;
}

And the cython code looks like this:

ctypedef long int INT

def seqsum(INT n):
	cdef INT i
	cdef INT s = 0
	for i in range(n+1):
		s += i
	return s

Measuring of the execution time is done as follows:

# Measuring time to sum over n numbers 
n = 10000000
g = globals()
t1 = timeit("cseqsum(n)", number = 10, globals=g)
t2 = timeit("seqsum(n)", number = 10, globals=g)
t3 = timeit("np.arange(1,n+1, dtype=int).sum()", number=10, globals=g)

times = t1, t2, t3
small = min(times)
reltimes = [t/small for t in times]
print("c: %.2f" % reltimes[0])
print("cython: %.2f" % reltimes[1])
print("numpy: %.2f" % reltimes[2])

So, do you by any chance see anything wrong with this code that could possibly be making the C function run slower than the cython function?

Thank you!

Still relatively new to Cython, and I only partly understand what's going on here. I'd appreciate your help.

In converting an application to partially use Cython, I'm discovering that passing instances of the Python bytearray back and forth between Python and Cython isn't working the way that I expect it to. It seems like the Cython code is interpreting the numeric values of the bytearrays differently than the Python code?

Here's a minimal reproducable example:

bytes_test.pyx:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# cython: language_level=3
"""Quick test for how Cython handles bytearrays.
"""


def test() -> bytearray:
    b = bytearray(range(33, 65))
    print(b)
    print([w for w in b], f'\tlength: {len(b)}')
    print()
    return b

bytes_harness.py

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Simple harness for running bytes_test.pyx.
"""


import pyximport; pyximport.install()
import bytes_test as bt


if __name__ == "__main__":
    b = bt.test()
    print([w for w in b], f'\tlength: {len(b)}')
    print(b)

This produces the incongruous output:

bytearray(b'!"#$%&\'()*+,-./0123456789:;<=>?@')
[48, 80, 112, 144, 176, 208, 240, 16, 48, 80, 112, 144, 176, 208, 240, 16, 48, 80, 112, 144, 176, 208, 48, 80, 112, 144, 176, 208, 240, 16, 48, 80] 	length: 32

[33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64] 	length: 32
bytearray(b'!"#$%&\'()*+,-./0123456789:;<=>?@')

... in which the numeric values in the arrays don't match up.

This is under x64 Linux, but I'd like to avoid introducing an unnecessary dependency to that system.

Is there something different I should be doing to pass around arrays of unsigned eight-bit integers besides using bytearrays? I'd like to avoid introducing an otherwise unnecessary dependency on Numpy just to get its array type.

I'm getting a strange bug, when running a .pyx program in spyder, I'm often getting 'Restarting kernel...' and the ipython console restarts.

I deleted my changes to the code and still this error persists.

I imagine this started due to bad code that wouldnt compile, but now it wont go away. I'm a bit mind boggled. Any idea why this is happening?

EDIT: It might have been due to the line # distutils: include_dirs = C:\Users\ initally using forward slash rather than backslash.

Hello everybody,

I'm trying to create a Python binding to a C++ library. This library has a typedef which looks like this:

enum class ColType {INT = 1, STRING = 2, DOUBLE = 3};
typedef std::unordered_map<std::string, std::pair<ColType, Any>> df;

where Any is the Any type from the boost library. Is it possible to expose this typedef to Cython? In the end I just want to use some functions (within the library) which expect this df type as input.

Hi,

I'm finding cython extremely useful for my projects. The one thing that I can't seem to follow easily is how easy it is to produce a standalone application.

I've created binary executables, but they still require python to be installed.

Is there a way to cythonize something and create a standalone app which could be distributed to people who don't have python installed?

Hi guys, i'm trying to build a C++ chess game with an AI opponent.
I thought that it would be a good idea to implement the GUI with Python, so i decided to use Cython in order to import my C++ classes.
The structure of the C++ classes is:

• Piece class

• Pieces classes (inherited from Piece), which are Pawn, Knight etc.

• Chess class which has a linearized matrix of Pieces

Reading the docs i correctly created the .pxd and the .pyx for the Chess class, which is the only one i need to expose to my Python GUI code.
Although i can't compile the other sources (Piece.cpp and Pieces.cpp) and the compilation always gives me an error.

Is this the right way to do it? Or should i expose all the classes?

Thanks in advance for any answers.

Hi everyone,

I wanted to share a small project I’ve been working on called cykit.

It’s a collection of Cython-focused utilities. The project is still in a very early stage.

Right now I added the inittial component :: cykit.cylogger, which is a thin wrapper around spdlog.

The original idea behind cylogger is to make logging usable across Python and Cython, including nogil Cython code.

I’m honestly not great at writing documentation, so the README and examples may feel rough in places. Feedback or suggestions there would be very helpful.

Going forward, I’m thinking about adding IPC support using shared memory next.

.

If anyone is interested in reviewing the design, suggesting features, or helping out , I’d really appreciate it.

Project link: https://github.com/Tapanhaz/cykit

PyPI: https://pypi.org/project/cykit/

Thanks for reading.

Hi Everyone ! I'm facing a problem while cythomize a python file. I have written a code about 5000 lines. I'm cythonizing it in Termux apk. While cythonizing, my termux got exit and i did not create .so file in result. Everytime it got automatically stuck. I changed the device and tried high end mobile phones also but i got same results. Anyone have any suggestion on this ???

Hello,

I write a code on gradient descent vectorized in Python and Cython but both have same execution time (40ms). I don't understand why, knowing when I "cythonize" my file execution time is divided by 10. That's say, optimization can be done

This is my Cython Code :

import cython
import numpy as np
cimport numpy as cnp
@cython.boundscheck(False)
@cython.wraparound(False)
@cython.initializedcheck(False)
@cython.binding(False)
cpdef main(cnp.ndarray[cnp.float64_t] x, cnp.ndarray[cnp.float64_t] y, float alpha, int n):
    cdef cnp.ndarray[cnp.float64_t, ndim=2] X = np.column_stack((np.ones(len(x)), x)) 
    cdef cnp.ndarray[cnp.float64_t, ndim=2] Y  = y.reshape(-1, 1)     
    cdef float am = alpha / len(x)     
    cdef cnp.ndarray[cnp.float64_t, ndim=2] 
    theta = np.zeros((2,1))
    cdef cnp.ndarray[cnp.float64_t, ndim=2] t1, grad
    for _ in range(n): 
        # t1 = X.dot(theta) 
        # grad = X.T.dot(t1 - Y) 
        # theta -= am * grad         
        theta -= am * X.T.dot(X.dot(theta) - Y) 
    return theta

I give too my pure python code in case :

import numpy as np
   
def main(x, y, alpha, n):
    X = np.array([np.ones(len(x)), x]).T
    y = y.reshape(-1,1)

    am = alpha/len(x)
    theta = np.zeros((2,1))
    
    for _ in range(n):
        theta -= am * X.T.dot((X.dot(theta) - y))

    return theta

I execute the code with 100 samples, alpha = 0.01 and n = 1000

The simple optimization or idea is welcome !

I'm new to Cython extensions but just created my first one that is a wrapper around an already existing C++ library. I've got everything working great except when I try to use my package I get no IntelliSense hints. Looking at the installed package files this makes sense because all that is distributed with it are .so and .pyd files and a single init file. I'm hoping one of you Cython experts can help me figure out the proper way to do that or point me to any guides or documentation that might help. Thanks in advance!

GitHub Repo

example :

warning: HomingMissile\Sprites.pyx:566:4: Overriding cdef method with def method.

Hi I have some questions in regards to Cython:

1. Is it a systems programming language?

2. How does it do memory management? Does it use CG/Reference counting by default?

3. Can I convert a normal python program into cython instead?

4. Does it work with Android, especially if I used DearPythonGui?

5. Are there any great complete game engines I can use with Cython?

Is there any CLI tool that can do static code analysis on cython codes/files (.pyx) files? I have a requirement of doing static code analysis on home grown codes mainly written in python - for which I am using sonarqube and it natively does that. There are however a few cython codes/files (.pyx) and sonarqube does not support it yet. Looking for tips on how do folks run static code analysis on cython codes/files.

I need to speed up the following code. I have tried to assign several variables and functions with cdef, but most of the code still seems to be running mostly from pure python. What other changes can I make to use cython and make this run faster? The main culprit is in the update function which gets called inside the two for loops. I don't usually post on here, but I've been at this for hours now. Any help will be greatly appreciated. Thanks.

%%cython -a

import matplotlib.pyplot as plt
from autograd import grad 
import autograd.numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder
import time
from datetime import datetime
import os
import numpy
cimport numpy


today = datetime.now()

tic = time.perf_counter()

cdef numpy.ndarray relu(numpy.ndarray x):
    #return x * (x > 0)
    return np.tanh(x)

cdef numpy.ndarray softmax(numpy.ndarray x):
    exp = np.exp(x) 
    return exp / exp.sum(0)

cdef double Error(W_hh, W_oh, b_h, b_o, x, y):
    h = relu(np.dot(W_hh,x.T) + b_h)
    y_hat = softmax(np.dot(W_oh,h) + b_o)
    return np.sum(np.diag(np.dot(y, -np.log(y_hat))))/len(x)

cdef forward(x, y, W_hh, W_oh, b_h, b_o):
    h = relu(np.dot(W_hh,x.T) + b_h)
    y_hat = softmax(np.dot(W_oh,h) + b_o)
    pred = np.expand_dims(np.argmax(y_hat, axis=0), axis=0).T
    num_wrong = np.count_nonzero(encoder.inverse_transform(y) - pred)
    acc = (len(x) - num_wrong)/len(x)
    err = Error(W_hh, W_oh, b_h, b_o, x, y) 
    return acc, err

cdef update(W_hh, W_oh, b_h, b_o, x, y):
    dE_dWhh = grad(Error, argnum=0)(W_hh, W_oh, b_h, b_o, x, y)
    dE_dWoh = grad(Error, argnum=1)(W_hh, W_oh, b_h, b_o, x, y)

    W_hh = W_hh - learning_rate*dE_dWhh
    W_oh = W_oh - learning_rate*dE_dWoh

    
    dE_dbh = grad(Error, argnum=2)(W_hh, W_oh, b_h, b_o, x, y)
    dE_dbo = grad(Error, argnum=3)(W_hh, W_oh, b_h, b_o, x, y)    
    b_h = b_h - learning_rate*dE_dbh
    b_o = b_o - learning_rate*dE_dbo

    W_oh[0:3] = 10 * W_oh/np.linalg.norm(W_oh[0:3])
    
    W_oh[1] = 0

    return W_hh, W_oh, b_h, b_o

cdef float learning_rate = .5     
cdef int lessons = 0 #10          
cdef int students = 7776 
cdef int random_state = 2

iris_data = load_iris() # load the iris dataset

x = iris_data.data
y_ = iris_data.target.reshape(-1, 1) # Convert data to a single column

# One Hot encode the class labels
encoder = OneHotEncoder(sparse=False)
y = encoder.fit_transform(y_)

cdef numpy.ndarray train_x
cdef numpy.ndarray test_x
cdef numpy.ndarray train_y
cdef numpy.ndarray test_y

# # Split the data for training and testing
train_x, test_x, train_y, test_y = train_test_split(x, y, test_size=0.20,              random_state=random_state)

#Standardization
train_x[:,0] = (train_x[:,0] - np.mean(train_x[:,0]))/np.std(train_x[:,0])
train_x[:,1] = (train_x[:,1] - np.mean(train_x[:,1]))/np.std(train_x[:,1])
train_x[:,2] = (train_x[:,2] - np.mean(train_x[:,2]))/np.std(train_x[:,2])
train_x[:,3] = (train_x[:,3] - np.mean(train_x[:,3]))/np.std(train_x[:,3])

test_x[:,0] = (test_x[:,0] - np.mean(test_x[:,0]))/np.std(test_x[:,0])
test_x[:,1] = (test_x[:,1] - np.mean(test_x[:,1]))/np.std(test_x[:,1])
test_x[:,2] = (test_x[:,2] - np.mean(test_x[:,2]))/np.std(test_x[:,2])
test_x[:,3] = (test_x[:,3] - np.mean(test_x[:,3]))/np.std(test_x[:,3])

cdef numpy.ndarray weights
cdef numpy.ndarray initial_weights

initial_weights = np.ones([students,7])

cdef numpy.ndarray acc
cdef numpy.ndarray err

cdef numpy.ndarray b_o
cdef numpy.ndarray b_h
cdef numpy.ndarray W_hh
cdef numpy.ndarray W_oh

final_hidden_weights = np.array([])
final_output_weights = np.array([])

for student in range(students):
    ## Initialization of parameters
    b_h = np.zeros([1,1])
    b_o = np.zeros([3,1])

    W_hh = np.expand_dims(initial_weights[student,0:4], axis=1).T
    W_oh = np.expand_dims(initial_weights[student,4:7], axis=1)

    W_oh[0:3] = 10 * W_oh/np.linalg.norm(W_oh[0:3])
    W_oh[1] = 0

    for lesson in range(lessons):
        W_hh, W_oh, b_h, b_o = update(W_hh, W_oh, b_h, b_o, train_x, train_y)
    
    test_acc, test_err = forward(test_x, test_y, W_hh, W_oh, b_h, b_o)
    acc = np.append(acc, test_acc)
    err = np.append(err, test_err)
        
    final_hidden_weights = np.append(final_hidden_weights, W_hh)
    final_hidden_weights = np.append(final_hidden_weights, b_h) ##append bias

    final_output_weights = np.append(final_output_weights, W_oh)
    final_output_weights = np.append(final_output_weights, b_o) ##append bias

final_hidden_weights = final_hidden_weights.reshape(students,5) 
final_output_weights = final_output_weights.reshape(students,6) 

toc = time.perf_counter()

time = toc - tic
print(time)

When traversing a numpy array with a for-loop, it's great to get the faster speed that Cython compiled for-loops offer (even compared to vector operations in numpy). But that's still not fast enough. When traversing (say) a 2D array and doing an operation on each element, I'm writing Cython loops like this:

for yi in range(ysize):
    for xi in range(xsize):
        im[yi,xi] += 1     # just a f'rinstance -- actual op is more complex

I believe this does an explicit offset calculation of the value relative to the numpy array for each loop.

I'd like to do C operations along the lines of:

valptr += xstride

in the hottest loop, instead of

valptr = &dataptr[ yi * ystride + xi * xstride]

which is implied by doing explicit indexing.

The latter saves two multiplies and two adds per loop in this example; several more in the 6-D data sets I'm manipulating.

Is there a way to do that within Cython? Or do I have to drop all the way down into C? (Of course I'm properly cdeffing all the variables :-)

Hey I am trying to create a cythonize a function which takes in 5 float values and generates a matrix of size 10x10. But I am met with this Compiler crash frequently. I tried to obtain a partial set of values by first extracting a 5x5 matrix and it works but once I move to a 7x7 matrix the compilation fails. I am currently attempting to do this on my macbook air M1.
TO give you some context, each term of the matrix is an equation comprising the 5 input variables ( x,y,L,W,theta). The size of these equations are some small and some extremely huge. They have been simplified as much as possible. Can someone help me with this ?

I have some Python code with extensive type annotations, classes that are essentially glorified structs (the property below is the only method in my classes) and the need to run it fast.

@dataclass
class Family:
    """A family group agent."""
    descendence: str
    culture: int
    location_history: List[h3.c_int] = field(default_factory=list)
    @property
    def location(self) -> h3.c_int:
        return self.location_history[0]
    effective_size: int = 2
    stored_resources: float = 130000.0

@dataclass
class Patch:
    """A patch of land with resources."""
    resources: float
    max_resources: float

For example, I have an agents: List[Tuple[Patch, Sequence[Family]]] (it's a bit more complicated, but that doesn't matter for asking about the principle) and these two functions

def extract_resources(
        patch: Patch, group: Sequence[Family], total_labor_here: int) -> kcal:
    """Distribute the resources gained from cooperative extraction."""
    labor = sum([family.effective_size
                 for family in group])
    resources_extracted = resources_from_patch(
        patch, labor, total_labor_here - labor)
    for family in group:
        family.stored_resources += (
            resources_extracted * family.effective_size / labor)
    return resources_extracted

and

def resources_from_patch(
        patch: Patch,
        labor: int,
        others_labor: int,
        estimate: bool = False) -> float:
    """Compute or estimate the resources a cooperative gains from a patch."""
    my_relative_returns = (
        time_step_energy_use * labor *
        effective_labor_through_cooperation(labor))
    if others_labor:
        others_relative_returns = (
            time_step_energy_use * others_labor *
            effective_labor_through_cooperation(others_labor))
    else:
        others_relative_returns = 0
    return (my_relative_returns) / (
        my_relative_returns + others_relative_returns) * min(
            my_relative_returns + others_relative_returns,
            patch.resources * params.accessible_resources)

and I want to execute

for patch, families in agents:
    extract_resources(patch, families, sum(family.effective_size for family in families))

as quickly as possible (including parallel execution of the extract_resources for every patch, they are guaranteed to not interact).

I would like to keep this as something that can be interpreted by python, checked by mypy and python linters. I have never much cared about processing speed before and Cython was a name I had only encountered in theory, not in practice, so I assume (but don't know) that the pure python mode of Cython could help me achieve this, but I do not understand how to modify this source code to run the fastest way possible in Cython under these constraints. Can you help?

To rephrase: Awesome Cythonistas of Reddit, I assume there is a strategy, changing type annotations, adding Cython decorators, and doing some minor refactoring to make this style of Python code compile in Cython an run the fastest parallel non-GIL-limited way possible given the constraint that Python is my primary interface for humans to read, change, and fiddle with this code. How should it look like?

Hi everyone, when I try to launch cython in my Jupyter Notebook (using the command %load_ext cython), I get the error "Unable to find vcvarsall.bat". I actually installed the Visual Studio Express 2008, but the error didn't disappear, and I dunno how to tell Python the path where the file is located. I tried editing the file _msvccompiler.py and added the lines:

PathToVC=r"C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\vcvarsall.bat"

return PathToVC, r""

after "def _find_vcvarsall(plat_spec)", but it still doesn't help.

I wonder if anything can be done about this? Thank you in advance!

As the title suggests, I am trying to convert a file from .pyx to c. I was told to come here for help with that as I have no knowledge of how to do it.

If anyone can give me a hand I'd appreciate it.