Webhooks not working

[pedalermike]

I have a bunch of triggers + webhooks configured to issue external commands. They were reliably working up to this morning. I've checked the webhook URL and when I process it manually, the correct action is activated. It seems that the trigger to webhook action is no longer working. Is there a debug utility or log file I can use to figure out where the breakdown is?

Update: None of my triggers are working - this big problem as I'm using those to trigger other services via webhooks. I tried a test using IFTTT to watch for the trigger event and then issue the webhook, the response time is far from ideal but it did work. I have ~70 different triggers defined and it would be a pain to migrate them to IFTTT...

[pedalermike]

bump!

[abachman]

Hi pedalermike,


Do you have logging capability on the service that's expecting to receive the webhook requests? We haven't made any changes to our infrastructure that would change our ability to send requests, but if they all stopped working at once and they're all pointing at the same URL / domain, it's possible your hosting provider is doing something (starting Friday) to block or filter requests from IO.

We're looking into adding some simple request debugging (e.g., reporting the response status of the last webhook request for a given trigger), but that will take some time.


- adam b.

[pedalermike]

Hi Adam,

The requests are going to a third party server using a protocol similar to the REST API (api key...). To be sure I wasn't messing things up, I downloaded all my triggers with the REST API: https://io.adafruit.com/api/v2/pedalerm ... xxx...xxxx which gave me a nice readout that I harvested and then made pretty using a simple Excel macro. I clicked through each webhook and they all worked so its quite baffling..

Having some form of response log tied to each outbound webhook would be really nice...

Thanks
Mike

[pedalermike]

UPDATE:

I created a new trigger and instead of sending the webhook to (https://joinjoaomgcd.appspot.com/...) which I was using, I sent the webhook call to my IFTTT maker channel and from there sent it to https://joinjoaomgcd.appspot.com/. It worked..!

When I redirected the new trigger to https://joinjoaomgcd.appspot.com/, nothing happened...?

[pedalermike]

I'm checking with the service provider whether they are blocking requests. What IP address is used when sending out a webhook request?

[pedalermike]

Adam,

I created a new account last night on Adafruit IO and configured it with the same trigger + webhook that used to work. When I activated the trigger, the webhook push never showed up at the Join server or perhaps it was ignored. I think this eliminates the possibility that the issue is a configuration problem with my account.

Would you be able to ping joinjoaomgcd.appspot.com from the Adafruit broker (or system that pushes the webhook) and see if a connection is made? The IP address of the Join server resolves to 172.217.0.244.

Thanks Mike

[abachman]

good news (about debugging, at least):

I've deployed the webhook logging changes that we've been working on the last couple days, your triggers page should now show a "Last Run" timestamp and "Run Count" counter for every trigger. Because it's a new feature, everything starts at zero until the trigger is activated again. This is what the triggers page looks like now:

Screen Shot 2018-07-19 at 1.09.51 PM.png (29.71 KiB) Viewed 812 times

Additionally, all webhook triggers will have associated logging for their last most recent request. If everything worked fine, it should look like this:

Screen Shot 2018-07-19 at 1.10.08 PM.png (90.89 KiB) Viewed 812 times

If the status came back with anything other than 200 you should see it, and if the webhook endpoint gave a response, you should see that as well.

ALSO, some maybe bigger more helpful stuff. For IO+ we've turned email triggers back on and we've lowered the webhook and email trigger timeout from 10 minutes to 1 minute. For Free accounts, email triggers are still off but the form is more clear about that.

We've also added better internal error tracking, so we should be notified sooner if webhooks are failing. In general, those errors are not high priority since the existence of remote servers and the validity of user-provided URLs is out of our control. In this case, though, we can keep an eye on it. Pinging the URL you gave from our servers worked fine, which says they're at least reachable over the network and typical ICMP requests are still received. Hopefully this logging will let us see what else is going wrong.


- adam b.

[pedalermike]

Thanks! This will be very helpful in trying to figure out what is going on...

[famkraus]

Hi Adam,

I configured a test trigger and ran the webhook action. I got the following 400 error (see attached). Any idea why the far end reports a malformed request with missing JSON elements? If I copy and paste the webhook into a brower, it works fine. What could be the formatting difference?

[famkraus]

Adam,

I spoke with the Join Server admin and he examined the logs. Apparently, the Adafruit broker is trying to send a POST request with some data where it should simply be doing a GET request. Do you know why that is?

Mike

[famkraus]

bump!

[abachman]

Hi famkraus,


Using HTTP POST when sending webhook data to an external receiving service is (as far as I know) a standard practice across the industry. For example, SendGrid https://developer.github.com/webhooks/ and GitHub https://sendgrid.com/blog/whats-webhook/ are two pretty significant developer-focused services that do the same. While our service is nowhere near as well documented as theirs, we also use POST requests since our webhooks include data in the body of the request. This doesn't mean it's a universal practice, though. We are unlikely to add an option to change the request type in the near future, but we can provide better documentation.

Also, I'm not familiar with Join Server / joinjoaomgc.appspot.com, it's possible if it's an open source project we could add a hook for Adafruit IO.

Two alternatives:

If you are able to use a service like Glitch, I built a tiny webhook receiver app to test our service. That glitch app can be remixed to proxy requests and translate HTTP POST + JSON request body to HTTP GET w/ URL params. If you're interested, I can add some code to that app to show what proxying requests through Glitch would look like. Honestly, it'd make a pretty good Learn guide, so I'll probably get to it eventually anyways.

I also am a big fan of https://ngrok.com/ which is kind of a global-to-local proxy / tunnel that can be used to connect an external URL (provided by ngrok, where you would receive a webhook request) to a service running on your local machine. I use it for Adafruit IO development to test external services that use webhooks to communicate--like Stripe, our payment provider--against the development version of our code.

Both of these are free services but require a bit of web backend programming.


- Adam B

[famkraus]

Hi Adam,

Thanks for the explanation. My takeaway from your note is that Adafruit IO has always used POSTs so the mystery of why the Join Server started rejecting them since a few weeks ago is open. Please do send me the info... to get around the impasse, I've reconfigured my triggers to loop through IFTTT which is a pain. I was considering setting up a proxy on my FreeNAS in the basement but that was looking to be a bigger job than I wanted..

[abachman]

the mystery of why the Join Server started rejecting them since a few weeks ago is open

Yeah, super weird. You and pedalermike are both using that service and it stopped working for both of you with no change on our end :(


- adam