How to daisy chain I2C with MiniGPS Moderators: adafruit_support_bill, adafruit

[gammaburst]

Hi danhalbert,
I wasn't using a BNO055 during my first investigation (with busio, erratic I2C, and the oscilloscope images). That problem occurs with various different I2C devices. Adding more devices makes the problem worse. Fiddling with voltage and pullups does affect the symptoms, but fiddling isn't the solution. I'm seeing a signal integrity problem on the falling edge of Titano's internal signal SCL_3V (see waveform below). I'm guessing the pin's drive strength "DRVSTR" mode is disabled. Enabling it may cure the ugly falling edge and solve the erratic I2C problem. Maybe! But I don't know how to enable DRVSTR in I2C mode.

I used the BNO055 during my second investigation (with bitbangio). That's when I noticed bitbangio seems to totally ignore I2C clock stretching. Increasing the timeout parameter had no apparent effect.

I don't have a different CIrcuitPython microcontroller that's conveniently available for testing.

In summary, I see three I2C problems with Titano + CircuitPython 7.0:
1. Poor signal integrity on signal SCL_3V. Its falling edge looks nasty. Enable DRVSTR?
2. bitbangio seems to not support I2C clock stretching. Fix the code?
3. To avoid unnecessary "no pull up found" errors, implement "Bus Clear" as described in I2C spec.

Comments/corrections welcome.

Oscilloscope image of Titano internal signal SCL_3V (yellow), and external SCL (blue):

[danhalbert]

Can you remove just one of the I2C devices at a time and see if one in particular causes a problem, but others don't?

Attached is a build with DRVSTR set to high (I think). Unzip it and load it and let us know if it makes a difference.

[sgalway]

Gamma is way beyond my expertise, but removing devices was the first thing I tried. See up the thread for that stage of troubleshooting.

The device I am attempting to build is for use on my boat. The i2c network is working with the 3.3v signal and pullups installed. Then when I key my VHF radio it crashes with the "no pullup" error. I am transmitting 25W and the antenna/coax is at least 6 ft away from the i2c network. The i2c network is working, but is far from stable.

[danhalbert]

Keeping RF out of the circuitry is a job. 6 feet is not that far from the antenna. The RF could be coming in via direct radiation or coupled through the power leads. Try putting everything in a grounded metal box, though you will have to expose the GPS antenna, of course. You may need to put chokes and/or bypass caps on the power leads. The unshielded/untwisted I2C leads are also nice receiving antennas.

[gammaburst]

Hi danhalbert,

The types of devices doesn't seem to affect the erratic I2C problem, it's the number of devices and interconnecting cables. For me, Titano busio I2C becomes somewhat erratic with 3 devices, and terrible with 5 devices. 5 devices somehow causes Titano to output only 5 SCL pulses per I2C transfer rather than 10.

Your titano-high-drvstr.uf2.zip significantly improves the SCL signal integrity problem, although it doesn't cure it. The internal SCL_3V waveform has improved although it's still not great. My busio I2C bus scan is now stable with 11 devices (that's all I have) using 50mm Qwiic cables. That ugly hump during the falling edge is now smaller, but it grows as I add more devices. I imagine that a few more devices could begin causing trouble.

I have not investigated the cause of the hump. Perhaps transmission line reflection. Why it causes Titano to freak out, I can only guess. The CPU reads SCL so it can do clock-stretching, so maybe the designers didn't build-in any noise immunity. The I2C bus is unterminated, so they should expect reflections. I'm just guessing here.

sgalway, you may want to try danhalbert's modified CircuitPython.

[sgalway]

Gamma,
I hate blindly trusting the results of others. :)

Do you know a resource where I might go to try to understand the information you are displaying in the image files?

[danhalbert]

@gammaburst - the STEMMA connector has level shifters to accommodate 5v I2C devices. I think you know that already. See the schematic here: https://learn.adafruit.com/assets/86212.

Have you tried feeding the chain from the SCL_3V and SDA_3V lines, bypassing the level-shifting transistors? I don't think that would necessarily make any difference.

You said you changed the STEMMA power jumper from 5V to 3.3V, right?

If you had another simpler CircuitPython device, I'd suggest you try the same chain on it, but you remarked you didn't.

The I2C bus spec max length is only a meter in "clean" conditions, and the STEMMA connectors introduce a lot of "bumpiness". You are using 100kHz clocking, I think. Unfortunately on the SAMD51 implementation, this is not lowerable below 95kHz or so, due to clock granularity issues. Otherwise I would suggest lowering it below 100kHz.

Thanks for the feedback on DRVSTR -- it looks like we should incorporate that change.

[gammaburst]

sgalway, I think danhalbert is an Adafruit CircuitPython developer.

Makes sense that the problem increases at 5V. The higher signal amplitude and higher pullup current tends to increase the amplitude of the reflection/glitch pulse.

I was thinking the same thing about the I2C bus length and distributed loads. This could be a unfortunate combination of (unnecessarily) fast fall time and an I2C controller that's unusually sensitive to a reflection/glitch shortly after the falling edge.

I noticed that it wouldn't let me go below 100 kHz. I tried 100, 200, and 400 kHz, but clock rate doesn't seem to affect the problem symptoms. Makes sense, the glitch always occurs regardless of clock rate.

I do have a couple other CircuitPython contollers such as an Arduino MKR Zero, but they're currently in use. I may be able to temporarily borrow one from its project.

I too thought about bypassing the MOSFET level shifters to see what happens.

I'm currently suffering from an acute lack of spare time.

[adafruit2]

if its a long chain, you may want to try a https://www.adafruit.com/product/4756 at the head of the chain

this device will clean up the signals a ton - it may be worth trying! what we're doing to i2c here is so out of spec :)

[gammaburst]

The glitch is not a transmission line reflection. I shouldn't have used that word. Apologies!

The falling edge glitch appears to be caused by the level converter MOSFET's turn-on delay. When the CPU initially pulls down SCL_3V, the MOSFET is not conducting so the CPU easily pulls down to ground. A few nanoseconds later the MOSFET begins conducting and discharges all the SCL bus capacitance into the CPU's pull-down transistor. That current is high enough to briefly yank the pull-down transistor out of saturation and up a couple volts, hence the glitch.

I can approximately reproduce Titano's busio scan malfunction and falling edge glitch waveform on SCL_3V by replacing my chain of I2C devices with a simple 200pF capacitor from SCL to GND. The CPU freaks out! It outputs only 5 clock pulses per I2C transaction, it "detects" non-existent I2C devices, and it pretty soon freezes/crashes (the CIRCUITPY folder stops responding). During this test Titano was configured with unmodified CircuitPython 7.0 and jumpered for 5V I2C.

If I instead connect that same 200pF capacitor from SCL_3V to GND, the capacitor simply slows the fall time with no glitch, and the CPU continues working fine.

This is feeling more and more like a design problem in the CPU's I2C controller.

Comments welcome!

[gammaburst]

Hi adafruit2,

I'll try an Adafruit 4756 although I don't see how it could help the problem we're discussing. UPS should deliver Friday.

Which I2C parameter is "so out of spec"? My 11-device chain is a couple mA high on pulldown current, I agree. The I2C spec says 50ns spikes must be rejected by a fast-mode I2C controller, so I think our ugly glitch is within spec. Anything else?

[danhalbert]

The bus capacitance is probably too high for spec, though we haven't measured it. In any case we will increase the drive strength: using non-high-strength drive may also be out of spec. Various microcontroller datasheets for instance discuss which pins are suitable for I2C and which are marginal, and we don't always choose the most suitable ones for various reasons.

[adafruit2]

i2c was not designed for long cables & off-pcb sensors. as you are adding more and more devices, each cable is adding capacitive and inductive load - the pullups are also in parallel, now <1K. what was a digital circuit is turning into an analog one :)

[gammaburst]

Please, we all should refer to the I2C spec when discussing what's right or wrong. The spec gives detailed measurable bus requirements and characteristics. Here's revision 7.0 (October 2021):
http://www.nxp.com/docs/en/user-guide/UM10204.pdf

I'm here because sgalway observed an unexpected malfunction with a relatively simple hookup of Adafruit components, and I easily reproduced that malfunction. I investigated briefly: the SCL_3V falling-edge glitch seems to be triggering the malfunction, however the I2C controller must include a 50 ns spike filter (spec Table 10) and that should reject the glitch. So why is Titano's CPU (or its firmware) going berserk? Is the spike filter missing? Is there another explanation? Wish I had more spare time to help debug this.

Reminder: sgalway and I triggered the malfunction with only three connected I2C sensors. I triggered berserk mode with one simple 200pF capacitor from SCL to GND. (Spec Table 11 says 400pF max, or even more by following paragraph 7.2.)

I'm sure Adafruit wants all their products to be reliable, easy to use and fun, so I suggest Adafruit investigate too.

[danhalbert]

Thanks a lot for your investigations and reading of the specs. I looked at the SAMD5x datasheet and looked at the code in more detail. The I2C peripheral says it has, quote:
"– Slew-rate limited outputs
– Filtered inputs"
This implies to me the spike filter is built in. There are no register controls to enable or disable these; there is not even further discussion in the datasheet of these features.

We are using the recommended register settings for other values, such as the hold times, though I can verify that these values are correctly set. If they were not, though, I think the waveform timings would be obviously off.

There is a lot of capacitance in the STEMMA QT connector assemblies. It would be interesting to repeat the test with short jumpers between the pins on the breakouts instead of the cables. (I am not asking you to do that :) ).

You mentioned: "During this test Titano was configured with unmodified CircuitPython 7.0 and jumpered for 5V I2C.". We don't recommend this. On the original PyPortal, the jumper was set by default to 5v, but that was not the best choice: we changed that on the Titano to be 3.3v, to match the typical use cases. I speculate that the PyPortal was designed before the STEMMA QT line of breakouts was completely set as a product direction (notice it has the larger JST connectors), and we wanted to handle existing 5v-only I2C breakouts, which are now less common.

In general, we have found that the older Atmel/MicroChip peripherals are pretty well designed. The SAMD21 works well, and the SERCOM peripheral designs were probably mostly transferred over to the SAMD51. The SAMD51's ADC and DAC have different features than their SAMD21 counterparts, and a lot more problems, several of which have not really been corrected in later revisions