With VR + VNF + L2 Network, the VMs in L2 Network cannot be reached from the Internet (But the opposite is possible)

[btzq]

ISSUE TYPE

• Bug Report

COMPONENT NAME

VR + VNF

CLOUDSTACK VERSION

4.19.1

CONFIGURATION

Advanced Networking
VPC
VNF + L2 Network

OS / ENVIRONMENT

SUMMARY

We have 1 VPC and 1 L2 Network connected together with a VNF (PFsense) in between.

Results Summary:

• Test 1: From L2 VM (10.26.8.230) to VPC VM (10.26.1.250) - PASS
• Test 2: From VPC VM (10.26.1.250) to L2 VM (10.26.8.230) - FAILED
• Test 3: From L2 VM (10.26.8.230) to Private Gateway VM (10.88.5.82) - PASS
• Test 4: From Private Gateway VM (10.88.5.82) to L2 VM (10.26.8.230) - FAILED

Background:
We’ve allowed any on the VNF firewall rules as attached in the screenshot below:

Traffic is allowed any on 10.26.1.254 interface

Traffic is allowed any on 10.26.8.254

We then run some tests.

Test 1: From L2 VM (10.26.8.230) to VPC VM (10.26.1.250)

Results: Ping and traceroute test from 10.26.8.230 to 10.26.1.250 works as expected. (PASS)

Test 2: From VPC VM (10.26.1.250) to L2 VM (10.26.8.230)

Results: Ping and traceroute test from 10.26.1.250 to 10.26.8.230 is not possible (FAILED)

Test 3: From L2 VM (10.26.8.230) to Private Gateway VM (10.88.5.82)

Results: Ping and traceroute test from 10.26.8.230 to 10.88.5.82 works as expected (PASS)

Test 4: From Private Gateway VM (10.88.5.82) to L2 VM (10.26.8.230)

Results: Ping and traceroute test from 10.88.5.82 to 10.26.8.230 does not work (FAILED)

We've been trying to debug Tests 2 and 4 for a while now with no sucess. Anyone has any ideas?
Or is Cloudstack designed not to allow this to be possible?

STEPS TO REPRODUCE

NA

EXPECTED RESULTS

Tests 2 and 4 should be able to work

ACTUAL RESULTS

Tests 2 and 4 does not work

[weizhouapache]

@btzq
it looks like a route to 10.26.8.0/254 is missing in the VPC VR.
you can retry aftering adding a route manually.

ip route add 10.26.8.0/24 via 10.26.1.254

[btzq]

Hey @weizhouapache , but this would require operator to manually intervene, right?
And if the router is destroyed (eg. CleanUp), this setting will dissapear, and it will need to be re-added manually.

Is there another way to do it?

[weizhouapache]

Hey @weizhouapache , but this would require operator to manually intervene, right? And if the router is destroyed (eg. CleanUp), this setting will dissapear, and it will need to be re-added manually.

@btzq correct.

Is there another way to do it?

no. currently static routes only applies on VPC private gateway.
https://cloudstack.apache.org/api/apidocs-4.19/apis/createStaticRoute.html

We have plan to extend it to support VPC and isolated networks.
probably next major release (4.21).

[btzq]

@weizhouapache i see, looking forward to this enhancement then!

Should i close this ticket? Or leave it open to track this request?

[weizhouapache]

@weizhouapache i see, looking forward to this enhancement then!

Should i close this ticket? Or leave it open to track this request?

you can keep it open.

btw: have you tested the manual static route ? does it work in your case ?

[btzq]

@weizhouapache
We tried your solution and it did not work.

After we've added the route, the VR is able to reach the destination, but source to destination still wouldnt work.

[weizhouapache]

@weizhouapache
We tried your solution and it did not work.

After we've added the route, the VR is able to reach the destination, but source to destination still wouldnt work.

What are the network acl rules of the vpc tier ? Allow all ?

[btzq]

Hi @weizhouapache , apologies ive missed your comment. Adding the manual static route still didnt work. We arent sure why... ACL Rules were Allow All to make sure ACL is not the issue, but it still didnt work...

Also, we had some growing requests from users where they want to implement a 'Hub and Spoke' architecture such as:

• https://cloud.google.com/architecture/deploy-hub-spoke-vpc-network-topology
• https://learn.microsoft.com/en-us/azure/architecture/networking/architecture/hub-spoke

And the 'Hub', needs to have some form of a VM Firewall (FortinetVMFW, PfsenseVMFW) to be the first point of contact to filter any incoming/outgoing traffic. To my knowledege, this is best done using L2 networks and not possible in VPC/Isolated Networks because Static Route is not supported in this network offering. But, our users want to have VPC/Isolated Network as the 'Hub' as they dont want to use L2 Networks.

I guess its what i have reported in this ticket. I really hope 4.21 resolves this at it would really help!!

Btw, is it possible to 'Force' a spoke VPC to use the Hub for its default gateway? (Eg. Internet access) using Static Routes on VPCs when it does become available in 4.21?