Wifi for the Mobile Blood Bank Buses

A few months ago network engineering was approached by the Blood Bank IT manager. He was looking for a way to provide wireless access to the mobile blood bank buses while they were parked at the MD Anderson blood bank so they could upload data once they return rather than having to store the data on a laptop and bring it into the blood bank building to upload the data with a wired connection.

Originally, the plan was to install Cisco access points in the parking lot that could be used to provide wireless access to the buses as they sat in the parking lot. Eventually, delays getting the access points deployed provided us with an opportunity to provide a solution that is significantly better than providing wireless access in the parking lot.

We came up with a way to provide a mobile wireless solution that provided institutional wifi access wherever the blood bank buses went. This was accomplished with two pieces of equipment. A Cisco Aironet 3502 or 1142 access point and a Cisco 819 series router with 4G LTE functionality built into it. The cell modem configuration on the router was fairly straight forward and only required a few simple commands. The trickiest part was configuring the cellular profile, bet even that wasn't overly difficult. The configuration documentation can be found here.  Once the access point is connected it connects to a wireless controller in the standard manner assuming it can reach the controller through the network.  For us, once the VPN tunnel is established the access point is allowed to reach institutional resources as if it were located inside the institution.

This is what the contraption looks like...

After testing for a month the system works well.  We have had a few issues with weak cell signal, but that is to be expected.  MD Anderson is planning on deploying these things in all the blood bank buses.  We will also be testing them with the remote blood bank units that perform on site blood draws.  The key difference with these is the lack of a bus.  The blood bank personnel go on location and set up inside a building.  A greater number of issues are anticipated for these tests since the routers will be deployed inside buildings that may have cellular connection issues.  I expect I will be updating this blog at some point or adding a new one once that testing is done.

It should be noted that, in addition to the 819/Access point combination with also tried a Cisco 1900 series router with an integrated wireless access point and a cell modem card.  This was abandoned for a number of reasons.  The router is quite large and takes up more space than the other two devices combined.  The cell connection took about fifteen minutes to come up.  Debugging revealed that the connection would repeatedly fail until it finally got a stable connection.  This happened regardless of where we tested it and it should be noted that we have an AT&T D.A.S. system deployed throughout the institution so we always have good cell coverage.  Finally, when the integrated access point was added to an AP group it rebooted and failed to come up.  This was problematic because we only wanted to provide on SSID to the blood bank buses.

In the future we are going to test some other solutions to see if there is anything better out there.  We plan to substitute the access point with an Aironet 600 series office extend access point.  Once configured for OEAP a 600 series access point will be able to connect to the institution without the hassle of a VPN tunnel.  Also, early in 2014 Cisco is supposed to release a version of the 819 router that has an integrated cell modem as well as an integrated access point.  We will be testing the new 819 as soon as we can get our hands on one.

CSCuc19950 - WLC(7.3) adds new mobility peers to all anchored ssids after reboot

For my first blog post I thought I would start with the reason I decided to start a blog in the first place.  A little known gem called CSCuc19950.  This bug bit me a couple of times during recent code upgrades and when I reported it to Cisco TAC I was provided with the following gem as an action plan...

"I would request you get in touch with you network team and verify the existence of the mapped controllers in the network and any changes made recently. In case you are the only person handling the devices then I would request you to remove the entry of the controllers that you know should not be there and save the configuration. Then monitor the devices and see if this repeats (very unlikely) to clear you doubts.

The possibility is that maybe there were configuration changes made on the controller previously and not saved so in case the WLC was rebooted the old entries have popped up."

In other words, tell your people to stop screwing up your WLC configurations.  Not one of Cisco's better responses, but after pointing out that I am the only one who makes these sorts of changes and indicating that I have spoken to other wireless engineers who have run afoul of this issue the TAC engineer did some more digging.  What he found was Cisco bug CSCuc19950.

Here is a screen capture from Cisco's bug search tool.  You will need a valid CCO login in order to view this for yourself:

Although undocumented, I believe this issue actually predates WLC code version 7.3.101.0.  I have experienced this sort of thing in the past and dismissed it as a faulty configuration without ever reporting it to Cisco and I suspect I am not the only one.  I have also spoken with other wireless engineers who reported seeing the issue prior to the 7.3 code release.

Here is what the issue actually looks like:

In testing with more that 20 Cisco 5508 wirless lan controllers I was able to create the behavior in three of them.  Interestingly, the issue occurred multiple times on these same three WLCs, but never on any of the others.