0

Adding a shared (ext?) module question
Moderators: adafruit_support_bill, adafruit

Please be positive and constructive with your questions and comments.

Adding a shared (ext?) module question

by clolsonus on Thu Feb 13, 2020 12:44 pm

I started following the documentation for adding a C code module to circuitpython. Things were progressing well until I realized the example only adds a module to the atmel-samd port. I have a teensy4.0 so I can't actually test run the example. I see the mimxrt10xx/Makefile is similar, but not entirely the same. The code I would like to bundle is not hardware/device specific. It is an attitude estimation filter that should run on any port or any hardware. I'd love to test it out on unix as well.

Is there a way to add a new shared module to the unix port (that is documented somewhere)?

I have a picture in my head of a $10 oshpark board + teeny 4.0 + adafruit imu + a few more bits == a circuit python based DIY flight controller for drones/uavs. I have a pretty high quality EKF we have used in many other projects, but it needs to run in native C++ for performance reasons.

Is there a single place I can add 'universal' c code modules? Or do I have to do this in each port individually and am I not able to do this in the unix port which doesn't seem to have the same shared-module setup?

Sorry if these are all dumb questions and documented elsewhere ... I have hunted around a fair bit and not turned anything up yet.

Thanks,

Curt. (U of MN UAV lab.)

clolsonus
 
Posts: 8
Joined: Thu Feb 13, 2020 12:19 pm

Re: Adding a shared (ext?) module question

by V2man on Thu Feb 13, 2020 1:15 pm

Last edited by V2man on Thu Feb 13, 2020 1:26 pm, edited 1 time in total.

V2man
 
Posts: 703
Joined: Mon Dec 03, 2018 12:38 am

Re: Adding a shared (ext?) module question

by clolsonus on Thu Feb 13, 2020 1:25 pm



Yes, that is exactly the guide I was working though which led me to the questions I asked above in my original post. Thanks!

clolsonus
 
Posts: 8
Joined: Thu Feb 13, 2020 12:19 pm

Re: Adding a shared (ext?) module question

by tannewt on Thu Feb 13, 2020 2:03 pm

Hi clolsonus, unfortunately we don't really use the UNIX port for new modules. It's largely to test the VM core only.

I'd recommend hopping into the Discord: https://adafru.it/discord We coordinate CircuitPython development in #circuitpython and are happy to help there. (I did the Teensy 4.0 board definition and you can mention me @tannewt there.)

Some other folks have been trying to link in a C++ library as well: https://github.com/adafruit/circuitpython/issues/2549

We're also close to merging in ulab which is a C library inspired by numpy: https://github.com/adafruit/circuitpython/pull/2583

tannewt
 
Posts: 1490
Joined: Thu Oct 06, 2016 8:48 pm

Re: Adding a shared (ext?) module question

by clolsonus on Thu Feb 13, 2020 3:29 pm

Hi tannewt,

Thanks for the clarification. It helps a bit knowing which rabbit hole might have a better chance of succeeding. I'll run down the path a bit further on my own and see if I can get a bit further before asking more dumb questions. And it's good to know there is a discord server out there as another option for connecting up with knowledgeable folks.

From a quick scan, it looks like the micropython approach to adding a C-code module is substantially different from circuitpython? Or am I drawing the wrong conclusion from the examples I've seen?

Thanks,

Curt.

clolsonus
 
Posts: 8
Joined: Thu Feb 13, 2020 12:19 pm

Re: Adding a shared (ext?) module question

by tannewt on Fri Feb 14, 2020 2:07 pm

Ya, we've diverged structurally from MicroPython in order to better split the Python -> C layer from the C layer. We also unified the Makefiles across ports as well.

tannewt
 
Posts: 1490
Joined: Thu Oct 06, 2016 8:48 pm

Re: Adding a shared (ext?) module question

by clolsonus on Fri Feb 14, 2020 4:45 pm

tannewt wrote:Ya, we've diverged structurally from MicroPython in order to better split the Python -> C layer from the C layer. We also unified the Makefiles across ports as well.


And it looks like v5.0 of the circuitpython code has diverged structurally from v4.0 with respect to C-code extensions. The tutorial no longer works and the differences are quite a bit beyond trivial (for me) to figure out. I made some additional headway today, but still getting a compile error:

CC ../../shared-bindings/mymodule/__init__.c
In file included from ../../shared-bindings/mymodule/__init__.c:3:
../../shared-bindings/mymodule/__init__.c:11:19: error: 'MP_QSTR_MyClass' undeclared here (not in a function); did you mean 'MP_QSTR_Flash'?
11 | { MP_ROM_QSTR(MP_QSTR_MyClass), MP_ROM_PTR(&mymodule_myclass_type) },
| ^~~~~~~~~~~~~~~
../../py/obj.h:168:56: note: in definition of macro 'MP_OBJ_NEW_QSTR'
168 | #define MP_OBJ_NEW_QSTR(qst) ((mp_obj_t)((((mp_uint_t)(qst)) << 3) | 0x00000006))
| ^~~
../../shared-bindings/mymodule/__init__.c:11:7: note: in expansion of macro 'MP_ROM_QSTR'
11 | { MP_ROM_QSTR(MP_QSTR_MyClass), MP_ROM_PTR(&mymodule_myclass_type) },
| ^~~~~~~~~~~
make: *** [../../py/mkrules.mk:55: build-teensy40/shared-bindings/mymodule/__init__.o] Error 1

Thanks,

Curt.

clolsonus
 
Posts: 8
Joined: Thu Feb 13, 2020 12:19 pm

Re: Adding a shared (ext?) module question

by tannewt on Tue Feb 18, 2020 6:38 pm

Whenever you see QSTR error first try `make clean` and then building again. The makefile rules around QSTR aren't perfect and you can get errors like yours as a result.

tannewt
 
Posts: 1490
Joined: Thu Oct 06, 2016 8:48 pm

Re: Adding a shared (ext?) module question

by clolsonus on Wed Feb 19, 2020 12:05 pm

tannewt wrote:Whenever you see QSTR error first try `make clean` and then building again. The makefile rules around QSTR aren't perfect and you can get errors like yours as a result.


I definitely don't fully understand the process of adding a c-code module yet, but I don't think it's that ... ? The missing symbol doesn't seem to be defined anywhere in the example code. Maybe there is some autogenerated code component or layers of macros? but then I don't know where those would go or how to unwind them and find them. And there's nothing in the failing example code that would #include them (that I can see.)

Let me ask this question from the other direction:

In the v5.0.0 beta code, can you suggest a relatively simple shared-module that I could look at as an example and pattern my work after? (I am working with the 5.0.0 because my device is a teensy40 and my understanding is that the v4.x of circuit python didn't have the teensy40 port yet, plus there are obviously structural/architectual changes in how this works with v4.x vs v5.0.0 so I might as well jump on the new system if I'm figuring things out anyway.)

Thanks,

Curt.

clolsonus
 
Posts: 8
Joined: Thu Feb 13, 2020 12:19 pm

Re: Adding a shared (ext?) module question

by tannewt on Wed Feb 19, 2020 2:56 pm

clolsonus wrote:
tannewt wrote:Whenever you see QSTR error first try `make clean` and then building again. The makefile rules around QSTR aren't perfect and you can get errors like yours as a result.


I definitely don't fully understand the process of adding a c-code module yet, but I don't think it's that ... ? The missing symbol doesn't seem to be defined anywhere in the example code. Maybe there is some autogenerated code component or layers of macros? but then I don't know where those would go or how to unwind them and find them. And there's nothing in the failing example code that would #include them (that I can see.)


There is autogeneration around all of the QSTRs which is why I'm suspicious of it.

clolsonus wrote:Let me ask this question from the other direction:

In the v5.0.0 beta code, can you suggest a relatively simple shared-module that I could look at as an example and pattern my work after? (I am working with the 5.0.0 because my device is a teensy40 and my understanding is that the v4.x of circuit python didn't have the teensy40 port yet, plus there are obviously structural/architectual changes in how this works with v4.x vs v5.0.0 so I might as well jump on the new system if I'm figuring things out anyway.)


I think bitbangio is probably the most straightforward: https://github.com/adafruit/circuitpyth ... /bitbangio Do you plan on making an object or just having functions on the module? _eve and _pixelbuf were recently added as well and may be good examples. Here is the _eve PR: https://github.com/adafruit/circuitpython/pull/2581

I'm happy to answer questions about the current code structure so feel free to ask here and/or on Discord (https://adafru.it/discord).

tannewt
 
Posts: 1490
Joined: Thu Oct 06, 2016 8:48 pm

Re: Adding a shared (ext?) module question

by clolsonus on Wed Feb 19, 2020 4:09 pm

tannewt wrote:Whenever you see QSTR error first try `make clean` and then building again. The makefile rules around QSTR aren't perfect and you can get errors like yours as a result. There is autogeneration around all of the QSTRs which is why I'm suspicious of it.

I think bitbangio is probably the most straightforward: https://github.com/adafruit/circuitpyth ... /bitbangio Do you plan on making an object or just having functions on the module? _eve and _pixelbuf were recently added as well and may be good examples. Here is the _eve PR: https://github.com/adafruit/circuitpython/pull/2581

I'm happy to answer questions about the current code structure so feel free to ask here and/or on Discord (https://adafru.it/discord).


Thanks! Pointing me at bitbangio definitely got me a few steps further down the path. Specifically I made modifications to .../py/circuitpy_mpconfig.h and .../py/circuitpy_mpyconfig.mk that paralleled the bitbangio example and I was able to get a good compile. (This is where the v5.0.0-beta5 structure totally diverges from the structure expected in the online tutorial.)

So the good news is I was able to flash the new .hex file to my teensy40 and it booted. I was able to connect with minicom -> /dev/ttyACM0 and run "import mymodule" successfully! Woohoo!

Next I ran the following which also looks promising:

>>> dir(mymodule)
['__class__', '__name__', 'MyClass']

Finally I went for the homerun and typed:

>>> m = mymodule.MyClass
>>> m.question

But the result came back as "<property>" instead of the expected "42". Similarly m.answer also returns "<property>":

>>> m.answer
<property>

I think I'm close, but still missing some small difference in the API to get back the actual value rather than some sort of <property> structure wrapper around my value?

Thanks again for your patience and help!

clolsonus
 
Posts: 8
Joined: Thu Feb 13, 2020 12:19 pm

Re: Adding a shared (ext?) module question

by tannewt on Wed Feb 19, 2020 4:19 pm

Currently properties only work on an instance of the class so do `m = mymodule.MyClass()` and then try it.

Please also post a link to your fork of the code with the changes if you have further questions.

Glad you are making progress!

tannewt
 
Posts: 1490
Joined: Thu Oct 06, 2016 8:48 pm

Re: Adding a shared (ext?) module question

by clolsonus on Wed Feb 19, 2020 5:18 pm

tannewt wrote:Currently properties only work on an instance of the class so do `m = mymodule.MyClass()` and then try it.

Please also post a link to your fork of the code with the changes if you have further questions.

Glad you are making progress!


Ok, hopefully this is the final report for now. I had missed copy/pasting a few lines of the example code in shared-bindings/mymodule/MyClass.c which led to a compile error. Initially I thought the issue was also due to v4.x versus v5.0 api changes (so I did some quick field triage that turned out to be wrong) ... it was just a simple copy/paste omission.

Also, as you gently pointed out, I had a python typo instantiating MyClass ... I forgot the () after MyClass.

With that code found and inserted in the proper place, and remembering my proper python syntax: I now have a working mymodule example running on the teensy4.0

I just did a local clone of the circuitpython repository with a local branch so I can't easily share the code changes with a link. If there is anyone that wants to update the online tutorial, I could share the specific details of things that have changed. I don't know the development culture here yet ... but if there is something specific I can do to help make this easier for the next person, just let me know what works best.

Thanks,

Curt.

clolsonus
 
Posts: 8
Joined: Thu Feb 13, 2020 12:19 pm

Re: Adding a shared (ext?) module question

by tannewt on Wed Feb 19, 2020 8:07 pm

Thanks Curt! Documenting the changes here would be great! We can then have a guide author update the guide based on it.

tannewt
 
Posts: 1490
Joined: Thu Oct 06, 2016 8:48 pm

Re: Adding a shared (ext?) module question

by clolsonus on Fri Feb 21, 2020 3:37 pm

tannewt wrote:Thanks Curt! Documenting the changes here would be great! We can then have a guide author update the guide based on it.


I don't think we want a pull request against the original circuitpython fork because these changes aren't destined for inclusion. Does this link work / make sense for documenting the differences and changes needed to get the mymodule tutorial working again?

https://github.com/adafruit/circuitpyth ... e?expand=1

There is some question about the changes I made to ports/mimxrt10xx/Makefile to add mymodule ... maybe there is a cleaner way to do that, but I can't quite see it if it's there. I don't recall needing to make any changes to the shared-module/mymodule code. I think most of the differences are in the build system infrastructure under py/circuitpy_*

Hope this helps the next person at least maybe a small amount?

Thanks! Now I can take a shot at what I'm really trying to do.

clolsonus
 
Posts: 8
Joined: Thu Feb 13, 2020 12:19 pm

Please be positive and constructive with your questions and comments.