Benchmarks for CircuitPython? Moderators: adafruit_support_bill, adafruit

[kevinjwalters]

Are there any benchmarks for CircuitPython? I noticed something in HackSpace recently (edition 34, page 67) and I was trying to reproduce the numbers in it which didn't look quite right to me for the two M4-based procesors mentioned, nRF52840 (64MHz) and SAMD51 (120MHz).

CircuitPython FP benchmark table from HackSpace Edition 34 page 67

hackspace-34-p67-benchmarktable.png.png (46.41 KiB) Viewed 194 times

Here's some very simple floating point tests, primary use here being to compare performance across boards. The speed difference here looks appropriate given the clock speeds of the two M4 boards. ALL running CircuitPython 5.3.1.

Feather M4 Express

Code: Select all | TOGGLE FULL SIZE

Adafruit CircuitPython 5.3.1 on 2020-07-13; Adafruit Feather M4 Express with samd51j19
>>>
>>> import simple_benchmarks
>>> simple_benchmarks.run()
platform: MicroChip SAMD51
version: 3.4.0
modules: {'simple_benchmarks': <module 'simple_benchmarks' from 'simple_benchmarks.py'>}
001AdditionA: 1.511477s (330802.226067Hz)
002MultiplicationA: 1.517326s (329526.972771Hz)
003DivisionA: 1.584962s (315464.997292Hz)
>>> simple_benchmarks.run()
platform: MicroChip SAMD51
version: 3.4.0
modules: {'simple_benchmarks': <module 'simple_benchmarks' from 'simple_benchmarks.py'>}
001AdditionA: 1.511183s (330866.622925Hz)
002MultiplicationA: 1.517409s (329509.019852Hz)
003DivisionA: 1.584792s (315498.876572Hz)


CLUE (alpha hardware)

Code: Select all | TOGGLE FULL SIZE

Adafruit CircuitPython 5.3.1 on 2020-07-13; Adafruit CLUE nRF52840 Express with nRF52840
>>>
>>> import simple_benchmarks
>>> simple_benchmarks.run()
platform: nRF52840
version: 3.4.0
modules: {'simple_benchmarks': <module 'simple_benchmarks' from 'simple_benchmarks.py'>}
001AdditionA: 3.610917s (138468.933105Hz)
002MultiplicationA: 3.594544s (139099.681854Hz)
003DivisionA: 3.706118s (134912.061691Hz)
>>> simple_benchmarks.run()
platform: nRF52840
version: 3.4.0
modules: {'simple_benchmarks': <module 'simple_benchmarks' from 'simple_benchmarks.py'>}
001AdditionA: 3.612694s (138400.876999Hz)
002MultiplicationA: 3.606740s (138629.317284Hz)
003DivisionA: 3.705397s (134938.311577Hz)

Circuit Playground Express (CPX)

Code: Select all | TOGGLE FULL SIZE

Adafruit CircuitPython 5.3.1 on 2020-07-13; Adafruit CircuitPlayground Express with samd21g18
>>>
>>> import simple_benchmarks
>>> simple_benchmarks.run()
platform: Atmel SAMD21
version: 3.4.0
modules: {'simple_benchmarks': <module 'simple_benchmarks' from 'simple_benchmarks.py'>}
001AdditionA: 9.364471s (53393.297195Hz)
002MultiplicationA: 10.113869s (49437.060356Hz)
003DivisionA: 13.884789s (36010.622978Hz)
>>> simple_benchmarks.run()
platform: Atmel SAMD21
version: 3.4.0
modules: {'simple_benchmarks': <module 'simple_benchmarks' from 'simple_benchmarks.py'>}
001AdditionA: 9.364391s (53393.750191Hz)
002MultiplicationA: 10.113922s (49436.793327Hz)
003DivisionA: 13.884777s (36010.656357Hz)

Code is in https://github.com/kevinjwalters/circui ... chmarks.py

[tannewt]

We don't maintain or track any ourselves. Generally, it can be a bit of a trap because there is always performance to gain at an increasing cost.

I've tweeted at Hackspace about how they got their numbers: https://twitter.com/tannewt/status/1315754129206042624

[kevinjwalters]

The ESP32-S2 numbers also looked odd in that HackSpace benchmark but I've only just got a board with that chip to puts its floating point through its paces:

FeatherS2 (ESP32-S2)

Code: Select all | TOGGLE FULL SIZE

sys.version: 3.4.0
sys.modules: {}
os.sysname: esp32s2
os.nodename: esp32s2
os.release: 6.0.0
os.version: 6.0.0 on 2020-11-16
os.machine: FeatherS2 with ESP32S2
001AdditionA: 1.4886s,1.3293s,1.4772s,1.3329s,1.4849s (14059Hz)
002MultiplicationA: 1.3424s,1.5280s,1.3663s,1.3788s,1.3277s (14403Hz)
003DivisionA: 1.9622s,1.9824s,1.9818s,1.9905s,1.9863s (10098Hz)
004ArchimedesPiA: 1.3958s,1.3141s,1.4156s,1.2751s,1.4064s (147Hz)


There's something curious going on with the addition and multiplication, it's almost like they are getting scheduled onto two different units running at different speeds.

micro:bit V1

Code: Select all | TOGGLE FULL SIZE

sys.platform: microbit
sys.version: 3.4.0
os.sysname: microbit
os.nodename: microbit
os.release: 1.0.1
os.version: micro:bit v1.0.1+b0bf4a9 on 2018-12-13; MicroPython v1.9.2-34-gd64154c73 on 2017-09-01
os.machine: micro:bit with nRF51822
001AdditionA: 20.8069s,20.8069s,20.8070s,20.8069s,20.8070s (961Hz)
002MultiplicationA: 22.4981s,22.4982s,22.4982s,22.4982s,22.4982s (888Hz)
003DivisionA: 35.0681s,35.0681s,35.0682s,35.0682s,35.0682s (570Hz)
004ArchimedesPiA: 18.7679s,18.7679s,18.7679s,18.7680s,18.7679s (11Hz)


micro:bit V2

I'd assumed the nRF52833 would have very similar performance to nRF52840 as they are both down as "64 MHz Arm Cortex-M4 with FPU" but apparently not

Code: Select all | TOGGLE FULL SIZE

sys.platform: microbit
sys.version: 3.4.0
sys.modules: {}
os.sysname: microbit
os.nodename: microbit
os.release: 2.0.0-beta.3
os.version: micro:bit v2.0.0-beta.3+d6c01d0 on 2020-12-21; MicroPython v1.13 on 2020-12-21
os.machine: micro:bit with nRF52833
001AdditionA: 7.6184s,7.6231s,7.6225s,7.6213s,7.6215s (2624Hz)
002MultiplicationA: 7.2355s,7.2386s,7.2354s,7.2372s,7.2362s (2764Hz)
003DivisionA: 9.2329s,9.2380s,9.2330s,9.2333s,9.2348s (2166Hz)
004ArchimedesPiA: 5.6066s,5.6113s,5.6098s,5.6072s,5.6111s (36Hz)


[kevinjwalters]

6.1.0-rc0 has made the FeatherS2 more consistent and faster! Or perhaps time.monotonic_ns() was buggy on ESP32-S2 6.0.0 and these have been fixed??

FeatherS2 (ESP32-S2)

Code: Select all | TOGGLE FULL SIZE

sys.version: 3.4.0
sys.modules: {}
os.sysname: esp32s2
os.nodename: esp32s2
os.release: 6.1.0
os.version: 6.1.0-rc.0 on 2021-01-06
os.machine: FeatherS2 with ESP32S2
001AdditionA: 0.9988s,0.9986s,0.9985s,0.9986s,0.9985s (20028Hz)
002MultiplicationA: 1.0196s,1.0194s,1.0193s,1.0193s,1.0194s (19619Hz)
003DivisionA: 1.5855s,1.5853s,1.5852s,1.5854s,1.5853s (12616Hz)
004ArchimedesPiA: 1.3531s,1.3614s,1.3529s,1.3530s,1.3699s (147Hz)
005MixedBigIntA: 2.7977s,2.9087s,2.7851s,2.9026s,2.8022s (1268Hz)


[kevinjwalters]

More benchmarking including over-clocked (200MHz) SAMD51 (M4) from EloquentArduino: Adafruit Blog: Feather M4 is a great chip for TinyML “Feather M4 Express is by far the fastest Cortex M4 board…”.