0

i2c lockup/freezing
Moderators: adafruit_support_bill, adafruit

Please be positive and constructive with your questions and comments.

i2c lockup/freezing

by madart9 on Wed May 22, 2019 10:01 am

Good morning,

I use an Adafruit MPR121 breakout boards to sense capacitance, but have been using them for years connected to a 3 foot 4-conductor headphone cable. The cable plugs into a main board with either an ItsyBitsy M4 or a 5V Pro Trinket. This has worked nearly perfectly.

I decided to use the MCP23008 i2c GPIO expander connected by 4-conductor headphone cable and now I am getting occasional i2c lockup. By this, I mean explicitly that SCL gets locked to 3.3V (on the ItsyBitsy M4 express with a 3.3V-5V level shifter)/5V (5V Pro Trinket) and SDA is LOW. This happens intermittently– anywhere between a few minutes and six hours. However, turning the power to all boards and Arduino off and on does not clear the problem! It requires disconnecting the cable a few times until something happens.

Can you help me recover the i2c connection while the Arduino continues to run? I would be extremely grateful. In searching the web (example citations at the bottom), there are a variety of comments, mostly suggesting manually clocking the SCL line until the confused i2c slave device clocks out the remaining bit or two. I have tried to implement this, but the results that other people find effective do not seem to be working.

I have tried to run this function at startup, with no success:
Code: Select all | TOGGLE FULL SIZE
void reseti2c() {
  int ntotal = 0;
  int nrunning = 0;

  pinMode(PIN_WIRE_SDA, OUTPUT);
  digitalWrite(PIN_WIRE_SDA, LOW);
  pinMode(PIN_WIRE_SDA, INPUT);
 
  pinMode(PIN_WIRE_SCL, OUTPUT);
  digitalWrite(PIN_WIRE_SCL, LOW);
  delayMicroseconds(3);
 
  while (ntotal < 1000 && nrunning < 20) {
    digitalWrite(PIN_WIRE_SCL, HIGH);
    delayMicroseconds(3);

    if (digitalRead(PIN_WIRE_SDA)) {
      nrunning += 1;
    }
    else {
      nrunning = 0;  // Always ends up at 20, so SDA does not seem to be held low by problematic MCP23008
    }
   
    digitalWrite(PIN_WIRE_SCL, LOW);
    delayMicroseconds(3);

    ntotal += 1;
  }
 
  pinMode(PIN_WIRE_SDA, OUTPUT);
  Wire.begin();
}


I have copied the code from here: http://www.forward.com.au/pfod/ArduinoP ... index.html to run on startup (and changed "SDL" and "SDA" to PIN_WIRE_SCL, PIN_WIRE_SCA), but this too does not work.

The only piece that I have found on other forums that I have not implemented is the ACK and NACK signals in the following site: https://bits4device.wordpress.com/2017/ ... -recovery/

In summary: i2c gets locked up intermittently (and apparently is very sensitive to EMI in long wires). I've assembled a bunch of boards and would love to be able to use them even though I know now that i2c over long wires is problematic. Do you know how to clear i2c lockup with MCP23008s? If not, do you know how to implement the NACK and ACK signaling while pulsing so that I might clear the signal correctly?

Thank you very much,
Arthur


Pre-emptive troubleshooting:
- I have multiple versions of both the ItsyBitsy M4 board and the Pro Trinket board. They are close to minimal, just being connected to power, i2c, and with a few GPIO outs to terminals. I am confident that this is not a hardware issue beyond my own bad choice of using i2c over long wires because it works the vast majority of the time and corresponds with other people's problems.
- The ItsyBitsy M4 board has a 5V regulator that powers both the ItsyBitsy and the i2c power line. The voltage drop over these cables is tiny and we're getting 5V at the MCP23008 and MPR121 boards. The regulator can put out an amp, so power should not be the problem.
- I don't have a logic analyzer, so I haven't been able to piece through clocking out the remaining bits, but it seems like the MCP23008 is not holding the line low as might be expected during the GPIO clockout in my reseti2c function.


Example references that I've found:
https://github.com/esp8266/Arduino/issu ... -319199959
https://www.microchip.com/forums/m175368.aspx

madart9
 
Posts: 28
Joined: Mon Jul 22, 2013 2:53 pm

Re: i2c lockup/freezing

by adafruit_support_mike on Wed May 22, 2019 1:29 pm

Try adding pull-up resistors to the SDA and SCL lines at the remote end of the cable.

My first guess is that you’re seeing trouble related to the parasitic capacitance has of the cable. The pull-up resistors have to bring the line voltage back to VCC when each I2C device releases a line, and the extra capacitance slows down the rising edges. Adding pull-ups that draw about 1mA when pulled low (3.3k for 3.3V systems, 4.7k for 5V) at every cable endpoint helps bring the voltage up faster.

You should be able to reset the bus by taking direct control of the SCK line and toggling it high and low about 16 times at 10Hz.

adafruit_support_mike
 
Posts: 59205
Joined: Thu Feb 11, 2010 2:51 pm

Re: i2c lockup/freezing

by madart9 on Tue May 28, 2019 12:05 pm

Good morning,

Thank you for your help! I've taken a while to reply to thoroughly test your suggestions.

First, they seem to have dramatically helped! I am still getting occasional errors, but so far clocking 16x seems to be working.

I have one final board that I connected via i2c over 4-conductor headphone jack that has a second ItsyBitsy M4, that seems to be causing more trouble. First, I had to follow advice on (I believe) the issue tracker of your GitHub to allow i2c slave connections (code included at the end). I purchased your level converters, but ended up using 2N7000 transistors and 4.7 kOhm resistors to wedge things next to each other. I've included the wiring diagram, and I apologize for its ugliness. In the long run, I'm going to switch back to your logic-level conversion boards, and I understand that you might not want to deal with this issue, but do you see any big mistake I've made? Will this be hurting the communication between boards?

Thanks much,
Arthur

Code: Select all | TOGGLE FULL SIZE
// Fix SAMD51 support
void SERCOM2_0_Handler() { Wire.onService(); }
void SERCOM2_1_Handler() { Wire.onService(); }
void SERCOM2_2_Handler() { Wire.onService(); }
void SERCOM2_3_Handler() { Wire.onService(); }
Attachments
Screen-Shot-2019-05-28-at-11.53.09-AM.png
Screen-Shot-2019-05-28-at-11.53.09-AM.png (21.13 KiB) Viewed 185 times

madart9
 
Posts: 28
Joined: Mon Jul 22, 2013 2:53 pm

Re: i2c lockup/freezing

by adafruit_support_mike on Tue May 28, 2019 9:05 pm

2N7000s are comparable to the BSS138 we use in our own I2C level shifter, so you shouldn't have any problems there.

You do need pull-up resistors on both sides of each mosfet: one to 3.3V and one to 5V.

adafruit_support_mike
 
Posts: 59205
Joined: Thu Feb 11, 2010 2:51 pm

Re: i2c lockup/freezing

by madart9 on Fri Jun 14, 2019 5:06 pm

Good afternoon,

Thank you very much for your help so far! I added resistors to the slave devices and this improved the error rate, but did not entirely solve the problem. I believe that part of the problem is that I've connected four 1-6 foot wires to a single i2c bus, which should be a substantial increase in capacitance.

I have a few more questions. I would love to be able to keep the majority of my design, but the current error rates over i2c are quite problematic. Here are solutions that I am considering. Would you be willing to suggest what is best?

- The MCP23008 seems to be the most finicky part over i2c. Another ItsyBitsy M4 seems more resilient to errors and the MPR121 capacitance breakout board has never really had problems. Should I switch to ATTinys/Trinket M0s or the like as slave devices rather than MCP23008s? Would they be less prone to errors?
- I have found references to the P82B96 online, suggesting that this should solve many of the capacitance problems (http://www.ti.com/lit/ds/symlink/p82b96.pdf). Would you suggest adding this chip in on both the master and slave sides even though it would be connected to four wires on the master side?
- i2C seems like such a simple mechanism for connecting different boards, especially thanks to your selling the MCP23008s. However, it's been surprisingly difficult to troubleshoot. In the abstract, should I be using a different communication protocol? I liked that I could use 4-conductor wires and that I could use an MCP23008 rather than a trinket on the other end so that I didn't have to program both sides, which is why I used i2c rather than serial.

Many thanks,
Arthur

madart9
 
Posts: 28
Joined: Mon Jul 22, 2013 2:53 pm

Re: i2c lockup/freezing

by adafruit_support_mike on Sat Jun 15, 2019 12:57 am

All signal protocols get complicated when you put more than a few inches of wire between the components. The wires themselves have parasitic inductance and capacitance that create a whole range of problems.

madart9 wrote:In the abstract, should I be using a different communication protocol?

If yout think debugging I2C at 100kHz is bad, you don't want to see what SPI does at a few megahertz.

If you have access to an oscilloscope, hook it up. Line problems are much, much easier to diagnose and solve when you can see what the signals are actually doing. Without that information, it's all guesswork. In most cases of blind transmission line debugging, the best you can do is an educated guess about the source of the problem after you've stumbled across something that mitigates the problem.

madart9 wrote:I have found references to the P82B96 online, suggesting that this should solve many of the capacitance problems

That's an I2C-friendly line driver, and yes, that should clear up your problems. You'll want one chip at each end of every connection.

adafruit_support_mike
 
Posts: 59205
Joined: Thu Feb 11, 2010 2:51 pm

Re: i2c lockup/freezing

by spoggles on Mon Aug 05, 2019 5:43 pm

Hello:

I was having a lockup issue with the MCP23008 and a BME680 when used together, but not separately.. I delayed the /reset pin on the MCP23008 upon power on and the issue seems to have disappeared. (I used a 0.0047 cap and a 100k resistor. Don't forget to place a diode between the cap and 3v3 supply across the 100k resistor. (Cathode to 3v3 and Anode to the cap to provide a discharge path to the cap so it is ready for the next power on)

Also I am using 2 p82b715s between 50ft of plane old 6 conductor telephone extension cable that I got from good will. I also tried it over 50<ft<100 of cat 5 wire and all seems good. The fact that you can use 600ohm pull ups on the loop side really increases the capacitance you can have on the line without integrating the SCL and SCK signals.

I would expect that it would work on 4 conductor phone extension line as well.

spoggles
 
Posts: 64
Joined: Tue Jun 02, 2015 12:32 pm

Re: i2c lockup/freezing

by spoggles on Mon Aug 05, 2019 7:50 pm

It would probably be more accurate to say I extended /RESET by 5mSec upon power on. Seems bringing /RESET and 3v3 supply on at the same time caused the problem.
The cut sheet indicates that the chip has its own internal POR, so I am having trouble understanding why POR is not taking care of this problem.

I have been trying 'stupid' things like removing USB power while still active and restoring power via USB randomly. Still working. SO FAR...

spoggles
 
Posts: 64
Joined: Tue Jun 02, 2015 12:32 pm

Please be positive and constructive with your questions and comments.