Thank you all for your great help!
I am always looking for alternatives :)
I had ordered a TXS0104 (people are claiming it is better for some protocols) It is pin compatible with TXB0104.
I believe you were right in that there was pull-ups in the modem that was too small and TXB0104 could not drive them to zero.
Seem however that TXS0104 manages to handle it. It looks perfect on the scope now, on both sides and it works flawless so far :)
I also read that it was possible to use diodes in one direction which was another alternative I was looking at.http://www.savagecircuits.com/forums/co ... 3V-Devices
since both TX and RX lines are high I would probably hurt the modem regardless if I connect anything directly to it.
Thanks again for all your help