AP's at remote faciltiy will not join local controller

Question

I currently have a WLC 4404 running 7.0.240.0. Management Interface IP is 10.128.55.10. AP-management interface is 10.128.55.15. Remote facility (facility A) has a VLAN created for wireless (455) and a VLAN 455 interface created with IP 10.133.55.2. Ports on the remote switch (switch B) have been added to VLAN 455 and AP's have been patched into these ports. The AP's are receiving IP addresses via a DHCP scope created specifically for this VLAN (10.133.55.0 /24) which is physically located at the local facility (facility B). The AP's are successfully receiving IP's from this DHCP scope. This switch is trunked to switch A in the same building. Switch A has a VLAN 455 interface configured with IP 10.133.55.1. VLAN 455 is permitted on this trunk port.

From the remote switches (both A and B), I am able to successfully source ping the AP-management interface (10.128.55.15) from the VLAN 455 interface (10.133.55.1, and 10.133.55.2). However, I am unable to successfully source ping the management interface (10.128.55.10) from the same VLAN 455 interfaces.

After working with Cisco TAC for 6.5 hours today, they say my wireless controller is possessed. I am not yet ready to accept a supernatural phenomenon as a reasonable explanation and was hoping someone out there can help!

UPDATE Took a known good AP down to the remote facility and it comes up on the correct VLAN, receives the correct IP and joins the controller. Brought back one of the non functioning AP's and plugged it in here. The AP joined the controller after downgrading its IOS from 15.2(2)JB to 12.4(23c)JA7. Should the AP's at the remote location be able to connect to the controller here and install the correct version of IOS? Or am I going to have to bring them all here for initial config prior to installing at the remote location?

UPDATE We have found a workaround to get the AP's to communicate. Cisco is saying it's a Microsoft issue with not pushing out option 43 properly. They also say there is no MTU issue while traversing the MPLS to the remote site. What we have done is disable the DHCP scope for the remote AP's and configured the remote switch (ios) as a DHCP server with option 43 pointing to the controller using hex not decimal. Once I did a shut/no shut on the interfaces connected to the AP's, they obtained an IP from the switch and joined the controller. Once joined, I removed DHCP from the switch and re-enabled the scope on our DHCP server. Bounced the AP's again and they pulled an address and joined the controller. Not a solution, but at least they're up and running.

Answer 1

I cannot tell you exactly what is happening without looking at the configs of the Switches and the WLC you have as I can think of a number of possibilities. For me, it'd be helpful to have that information but more so, the things that you've done. Basically, what are the items you've tried thus far?

SW-A: 10.133.55.1 <---TRUNK---> SW-B: 10.133.55.2

WLC-4404 10.128.55.10

First thoughts though are as follows:

1.) Who is able to ping the WLC? a.) no one = check the interface on the switch and the WLC interface config, any firewalls b.) same subnet, 10.128.55.0/24, but no other subnets = check routing c.) all subnets on one switch but not from another switch = check trunk on SW-A & SW-B

2.) AP and WLC are able to ping each other but AP not showing up in WLC a.) check DHCP has the proper option 43 (Vendor Specific Info) set which tells the AP what IP the WLC is. b.) check any firewalls inbetween c.) create new interface on WLC with the same VLAN subnet as the AP and test d.) enable ssh on the AP and do a show log to see if anything peculiar sticks out

3.) You've checked all possibilities and as the config stands everything should work a.) reload the switches, AP's, WLC b.) pull out the configuration and put the configuration back in specific to the issue c.) try a different interface on the switch for the WLC d.) try a different interface on the WLC (maybe create a new one) e.) get on Cisco.com and use the bugtoolkit to see what bugs are out there for your particular WLC version. f.) upgrade the WLC version a revision up and test.

I realize that some of my ideas are not really applicable, but I'm just throwing some ideas out there to maybe help give a new perspective on the problem. I've found just taking a break to think about something else and then coming back helps me.