Screwing the Cisco pooch

Let’s face it. Every techie enjoys a good stuff up, especially when it’s spectacularly performed by another individual. Today it was my turn.

I’ve been chipping away at getting recertified as a Cisco CCNA. This basically means messing about with switches, routers and the other bits and pieces that make up a network. Today I decided to take a switch to work so that I could experiment with syslog logging, broadcast storms and switch hardening.

I powered the switch up as soon as I got in and went off to get a coffee, the boot up process is somewhat slow. When I got back I decided to erase the existing configuration and start from scratch. For historical reasons Cisco has 3 different commands to achieve this erasure, I generally use one but this morning decided to mix it up a little, to get more familiar with the other commands just in case the exam hits me with them.

In my pre-caffeinated state I made a critical error, typing “erase flash:” instead of “erase nvram:”. What’s the difference? Well one is kind of like Windows “del config.txt”, the other is more like “format c:”. Yep, I completely erased the flash drive, including the switches operating system, thus completely bricking the device. In my shame I quietly powered it down and took it back to the car where it remained for the rest of the day.

Tonight it got better. I arrived home thinking it would only take a quick file transfer to get it all running again, how wrong I was. Cisco, in their infinite wisdom, for some reason decided to omit support for the TFTP protocol from the c3550 switch rom. After an hour or so of fiddling, chasing up the usual web misinformation, I eventually found a Cisco manual that outlined the recovery procedure - XMODEM.

For those of you born after 1990 or so, XMODEM is a really, really ancient way of copying files. Like it was used when bits were individually hand carved from phonelic resin and assembled in groups of three by strange wo/men in lab coats mumbling incantations. It was used when people had real phones, not these namby-pamby computer phone thingys, but real phones made from bakelite with cords and dials.

But wait, it gets better, XMODEM over the console port. Every Cisco device has a console port and in devices manufactured before 2010 (ish) it’s an RS-232 serial port running at 9600 baud. Seriously, you can catch those bits in your teeth as they flow down the wire.

A switch OS file is ~7M bytes. At NBN rates that amounts to a file transfer time of 2-3 seconds. Transfer time over XMODEM is > 2 HOURS! Think I need to break out my Duran Duran CD’s, big hair and leg warmers because I’m partying like it’s 1982.

For future reference, if you feel the urge to wipe an old Cisco switch like a 2950 or 3550, just don’t…

3 Likes

Welcome back! :stuck_out_tongue:
Ah Cisco. Silo development. You cannot be certain that different models running the same code version will behave the same way.

FWIW, I use the following to wipe switches: (I’m not CCNA, but I’ve been in this space for 10 years. No idea what CCNA wants you to do).

wr erase
del flash:vlan.dat

I know that the exams used to fail you if you ran wr mem instead of copy running-config startup-config so, I wouldn’t be surprised if wr erase is a no-no.

Hi,
the current CCNA prep book lists 3 options:

write erase
erase startup-config
erase nvram:

Normally I go for ‘erase start’ but I’ve been trying to alternate between them for muscle memory effect.

The rant above is largely due to my surprise that a device manufactured in 2004 and having numerous 100/1000 ethernet ports when faced with complete flash erasure can only be loaded via the console/serial port.

I thought about it over the weekend and I’m wondering if this indicates a little of the hardware design. Older devices like 2500/2600 routers were capable of using their onboard ethernet port to TFTP load an image, even when completely wiped. I wonder if the switches have a small ‘boot’ processor that is responsible for configuring a larger soft core processor in the FPGA/ASIC fabric. The small boot processor not being attached to or capable of driving the ethernet controllers, just enough to format and upload to the flash and then transfer the flash image during FPGA initialisation.

That would suggest that the switch image is part FPGA config file and part soft core processor machine code.

1 Like

I think there is truth to that - certainly on first boot after an IOS upgrade there are log messages about FPGAs being programmed.

1 Like

well done Steve but great recovery :slight_smile: