Sometimes the idea looks great on paper but doesn’t really work out when you try to configure it. And often, the only way to be sure is to break out the good old scientific method and try. So I tried. And it didn’t work, so I’m putting here in case you get a similar wild idea in near future.
The goal was to start with a primary VNET in Azure for some VMs. This network was going to act as a collection point for data coming in from a number of remote physical sites all over the world. In addition, some machines on the primary network would need to send configuration data to the remote sites. Ultimately, we were looking at a classic hub and spoke network design, with an Azure VNET in the center.
There are several ways you can do this using Azure networking, VNET peering between Azure VNETs, Site-to-Site (S2S) VPNs, and even ExpressRoute. ExpressRoute was off the table for this proof of concept, and since the remote sites were not Azure VNETs, that left Site-to-Site VPN.
The features you have available to you for Site-to-Site VPN depend on the type of gateway devices you use on each end for routing purposes. For multi-site connections, route-based (aka dynamic) routing is required. However, the remote sites were connected to the internet using Cisco ASA devices. The Cisco ASA is a very popular Firewall/VPN that’s been around since about 2005, but it only uses policy-based (aka static) routing.
So while we could easily use a static route to connect our primary site to any SINGLE remote network using the S2S VPN, we couldn’t connect to them all a simultaneously. And since we couldn’t call this a “hack” without trying to get around that very specific limitation, we tried to figure out a way to mask the static route requirement from the primary network. So how about VNET Peering?
VNET Peering became generally available in Azure in late 2016. Prior to its debut, the ability to connect any network (VNET or physical) required the use of the VPN gateways. With peering, Azure VNETs in the same region can be connected using the Azure backbone network. While there are limits to the number of peers a single network can have (default is 10, max limit is 50) you can create a pretty complex mesh of networks in different resource groups as long as they are in the same region.
So our theory to test was…. What if we created a series of “proxy” VNETS to connect to the ASA devices using static routing but then used the VNET Peering feature to connect all those networks back to the primary network?
We started out by creating several “proxy” VNETs with a Gateway Subnet and an attached Virtual Network Gateway. For each corresponding physical network, we created a Local Network Gateway. (The word “local” is used here to mean “physical” or on-prem if you were sitting in your DC!) The Local Network Gateway is the Azure representation of your physical VPN device, and in this case was configured with the external IP address of the Cisco ASA.
The we switched over to the VNET Peering configuration. It was simple enough to create multiple peering agreements from the main VNET to the proxy ones. However, the basic setup does not account for wanting to have traffic actually pass through the proxy network to the remote networks beyond. There are a couple notable configuration options that are worth understanding and are not enabled by default.
- Allow forwarded traffic
- Allow gateway transit
- Use remote gateways
The first one, allow forwarded traffic, was critical. We wanted to accept traffic from a peered VNET in order to allow traffic to pass through the proxy networks to and from the remote networks. We enabled this on both sides of the peering agreement.
The second one, allow gateway transit, allows the peer VNET to use the attached VNET gateway. We enabled this on the first proxy network agreement to allow the main VNET to direct traffic to that remote subnet beyond the proxy network.
The third one, use remote gateways, was enabled only on the main VNET agreement. This indicates to that VNET that it should use the remote gateway configured for transit.
One this was all set up on our first proxy network, it worked! We were able to pass traffic all the way through as expected. However, connecting to just one network with a static route was doable without all the extra things. We needed to get a second proxy and remote network online!
We flipped over to the configuration for the peer agreement to the second remote network. There we found we COULDN’T enable the “Use Remote Gateways” because be we already had a remote gateway configured with the first peering agreement. Foiled! 😦
Using a remote gateway basically overrides all the cool dynamic-ness (not an official technical term) that comes with VNET peering. It’s not possible with the current feature set of VNET peering to mask the static S2S VPNs we were trying to work around. It may be possible if we wanted to explore using a 3rd party VPN device in Azure or consider ExpressRoute, but that was outside of the scope of the project.
Still, it was fun to try to get it to work and learned a bunch about some new Azure networking features. Sometimes, the learning is worth loosing the battle.
7 thoughts on “The Network “Hack” that Wasn’t To Be”
Very good article. I’ve tried to do the same scenario without had read your post. I have the same requirement. I would like to use one proxy vnet for Cisco ASA (Static) and another proxy vnet for Dynamic S2S connections. And, is possible to use only one remote gateway config in the main vnet.
What is your suggestion to resolve this issue?
– Use ASAv as Virtual Appliance?
– Did you see Cisco released new firmware 9.8 for ASA (last week) that supports IKEv2 and VTI? I think this version/release resolve this issue (connect to Azure Gateway – Route based).
Yes, the updated ASA software may resolve the issue and there were also some virtual appliances that could be used. Since May, there have been some Azure updates that address the issue that I haven’t tested.
Hi, I just discovered the same issue – before finding this article! Would UDRs work rather than remote gateway configuration?
I don’t know for sure at this point. There were a lot of changes not long after I did this post. Did you try UDR with any success?
I haven’t tried this in practice but from my understanding of the hub and spoke topology I think it would work if you made the following changes:
1. Setup the VPN from your primary site to the hub without the additional proxy vnet (just hub to primary site via the gateway).
2. Enable allow gateway transit on the hub side of the peering configuration.
3. Enable allow remote transit on each of proxy vnets going to the hub in the peering configuration.
4. Keep the forwarding traffic enabled on all of the vnet peering configurations.
5. Add a route table and attach all of the vnet subnets to it.
6. Configure the routes to each remote site to pass through a virtual appliance with the default gateway address being the Virtual Gateway address that is deployed for each remote site (should be the first or second address handed out by Azure DHCP in the gateway subnet of the vNet (ex. 10.201.0.4)(you can run a traceroute to find the address).
Yes, what you describe should work now. At the time this post was written, several of the features required to do that didn’t exist.