My day with Azure: Introduction and Azure Application Gateway WAF issue

Hello all,

I realized that while working with Azure, I run into those issues here and there where even Bing cannot help you ( yup I did use Bing 🙂 ). You spend a whole day trying to troubleshoot then it turns out to be something so simple.

So I decided to start this series, where I document some of the issues I run into and what I did/do to resolve them. Hopefully these blogs end up helping someone who unlike me still has some hair left.

I will try to include screenshots if I can, otherwise please forgive me.

The first issue I will start with is something I faced with Azure Application Gateway (WAF). I spent a couple of hours troubleshooting, even involved Microsoft support but alas no one could help me.

This was my second time deploying an Azure Gateway WAF but the first time for a Windows environment. So my backend held two newly built Windows IIS Servers that had 2 sites on each, other than the default site that come by default on IIS.

I configured the bindings on my 2 sites properly by adding an https bind and attaching a certificate to it. see below. I did not touch the Default site since I thought I would not be using it at all.

 

I was able to successfully create the required listeners for port 80 and 443 on the front ends and also configure the rules. We required an end-to-end SSL, meaning that even the communication between the Gateway and the WebServers needed to be done on port 443. This I was failing to do!!!

If I did an SSL offload, meaning that I would receive connections on port 443 and then talk to the backend servers on port 80 that would work but port 443 (backend) kept giving me the dreaded 502 Gateway Error. Here are the few troubleshooting steps that I tried:

• First I opened the following ports on the Gateway subnet for the health probes: 65503-65534. The health probes were showing HTTP as healthy and HTTPs as unhealthy.
• Checked and rechecked and rechecked and removed the NSG rules on both the Gateway Subnet and the Webserver Subnet.
• I gave my backend server a public IP so that I can bypass the Gateway. I was able to successfully connect to those servers on https without any issues.
• Modified the listeners and the rules about a 100 times.
• Looked at the Gateway logs and if you have ever worked with an Azure application gateway, you know how hard that is. To give you an idea, I am not exaggerating when I say you have to dig into about 8 folders to get to that log file.
• Ran a network watcher on my backend servers and I was able to see a successful https connection coming into the server without any issues and handshake taking place.
• Involved Microsoft support who checked the gateway on the backend and said everything was fine.

After all the above this is the conclusion I came to, there is no issues with https on the two sites I have setup, the https probe is coming from the Gateway to the Server but the response is not 200 from the server causing the Gateway to throw a 502.

While thinking about this, I stumbled back to my Web Server and I noticed that in IIS, my default site only had port 80 configured on it and not 443. I said, lets add 443 and see what happens. I go the back-end health blade on the Gateway and WOOT WOOOT, below is what I see.

 

So it turns out, that by default the Gateway when looking at the backend servers to check if the ports are up goes to the / directory which is not under any site you created but under the Default site section and since I never created a 443 rule there, it never received a proper 200 response from the server on port 443. Below is how Default site’s binding must be configured.

 

And there you go, hopefully one day some of you reading this are facing the same issue and this blog helps you resolve it. Please comment below if you have any questions.