Cisco Certified Father

I write this to let off some steam, I write this to take a break and motivate myself.

Recently, as in 7 days ago I became a Father for the first time. It has brightened my life and my direction more than I could ever imagine. In the last 3 months of the pregnancy I have been studying to sit the CCIE Routing & Switching Written Exam. I need to re-certify my current Cisco certs and wanted to attempt this exam after failing it a few years back.

I enjoy sitting new exams so I can learn new topics and new skills. My ultimate goal is to sit the CCIE Lab to become CCIE certified. I will attempt this maybe in the next year or so I think.

I have until July to pass this exam, so I have time up my sleeve, although I am on four weeks leave and trying to study while looking after the new born. It is proving to be difficult, so I wanted to explain my past and now present study tactics I use.

During the last 3 months, I watched a CBT nuggets every morning on the train to work. On the way home I would read a CCIE Exam Cert guide. I used to watch videos for one week, books for one week and then labs for on week but this isn’t working this time.

I am pretty tired and was planning 2 hours a day during the week for study, currently I’m getting about 1 at the moment. Its not going to cut it and the exam is at the end of the month.

My new plan, I am doing the Boson Ex Sim CCIE written practice exams. These exams are really tough and I can’t seem to break through the 70% mark yet. If i do fail, Boson will pay for the resit, as they guarantee a pass. At this stage its looking likely they will fund a resit. I find myself wondering, do I know enough and these exams are too hard or I am way off?

I decided to go back to my old T-SHOOT simulator through Boson, to check my knowledge. I did the first four simulators and got them right. I have never had an issue troubleshooting complex networks.

So here I am, 3 weeks away from the exam and I am going to have to change things up.

To motivate myself, I will put my new plan and some advice I have heard once before here. It may help other people when sitting exams and also re-motivate me.

  1. Always read every possible answer before just diving into what you know is right
  2. Remove the answer you know for sure are incorrect, leaving better odds on a guess
  3. When clicking start on the computer at the exam, write down everything you can from memory. Especially subnet masks and wildcard masks. Also binary and hex tables so you don’t waste time on them.
  4. Take your time and don’t panic. Try and use real world examples when you were troubleshooting a protocol in the work place to help your logic.
  5. Never give up, learn your weaknesses and address them.

My current study plan is now –

  1. Small study till the end of this weekend (practice questions, chapters here and there)
  2. Starting Monday, One Video, One Lab and One full practice exam. After the practice exam, right down the troubling topics and use them the next day.
  3. For Monday I will be studying NBAR, IPv6 NAT and SDN.
  4. Never give up, learn your weaknesses and address them.

Hopefully Freddie (my son) can get some sleep and so can I to prepare for next weeks study. I am really one week behind and it was crazy to think I can study while looking after a newborn! But I am dedicated and I really want my CCIE. I want to really push myself to become one of the top engineers so I can one day either work for Cisco or become a Cisco teacher.

~Brad.

 

 

 

 

 

 

 

 

 

 

 

 

Advertisements

Passwords are important

I was sent to a Casino once. 7:00am start and doing remote hands work for our NOC. The Comms room was well hidden in a staircase. It was cool and dusty as I remember. 

Electronics don’t like dust.

I was there to replace hardware, in a redundant system, it can be swapped without impacting the operation of the switch. The time came and I placed my hands on the module, it’s latches were quite in place. I removed the module and inserted the new one.

Some lights flashed. I was done, or so I thought.

I received a phone call from the NOC, we can’t see the module you will need to log in. I didn’t have the password. 

Why didn’t I have the password?

They called the senior on call engineer and he arrived. He logged in and found the hardware I inserted was actually faulty as well.

We had been sent someone else’s failed module. 
When we left he said something to me which has stuck with me in my career and life.

Always be self sufficient. Bring everything you need and make sure you’re prepared. 

That was the last day I didn’t know a password. 

That was the last day I wasn’t prepared.

~Brad.

Network Engineers don’t wear suits?

I remember a 21-year-old CCIE telling me this. “Network Engineers don’t wear suits!” He had come out of a boot camp and had the world at his feet. The CCIE is the most sort after certification by business. It stands for Cisco Certified Internetwork Expert.

Although, instead of configuring routers and logging onto devices all across the world he was in customer meetings. Gathering requirements and trying to design networks that would allow better productivity and lower costs. His technical knowledge was in his head, but see they only looking at his face, his suit.

This is not what I was prepared for when I entered the world of networking as well. All I cared about was not having enough technical knowledge! I didn’t know the ins and outs of a 6500 switch or configuring MPLS! But I can speak to people. I can relate to people, I’m a people person!

Or so I thought…..

I had a bad experience once….

It put me off the suit….

It was a meeting where I wasn’t prepared and the good news I was giving the customer that nothing needed to be done, I thought it meant I really had not been doing anything the whole time I was there. I should have realised it was good news and put that spin on it. The meeting started, and my mouth opened. A few words came out something something ‘nothing is needed in your network’. Then….

Nothing.

I took a breath and closed my mouth. It wouldn’t open back up. I froze. How could I freeze? I was a new engineer worried about BGP , routing tables and not knowing enough technical information!,

I can talk to people. I can relate!

I got it wrong.

People skills are probably the most important part of a network engineers job.

That experience put me off for sometime. A long time actually. It wasn’t until I read the book ‘How to win friends and influence people’ when I started to learn the best way to deal with people. The book Who moved my cheese also made a big difference. Those books transformed my outlook and my life.

9 years has passed since I met that 21-year-old CCIE and 8 years have passed since that meeting.

The chapter I should have started 8 years ago starts today.

I should have worn the suit first.

~Brad.

4 am Phone Call – The answer is C.

Most IT related exams are multiple choice. I do remember my teacher once saying “When you get a call at 4 am in the morning, the correct answer isn’t C”.

Many years ago I was working for a company in Victoria, Australia. We had engaged an integrator to deploy a new WAN. The technology was GetVPN. This technology allows any site to talk to any other site via encrypted tunnels. These tunnels are not really tunnels but security associations between a source and destination. This technology is designed to be used over a MPLS network, a network which we call any to any connectivity. Traffic doesn’t need to pass through other sites, it can go straight to the destination.

This technology also relies on an underlying routing protocol to provide connectivity to all sites before the encryption takes place. I wont go into any more detail, but I will try to capture what happened that morning and the intense pressure we felt.

I don’t recall the date or day, but I do recall the time. 4 am.

My colleague was on call, and he was the lucky one to receive the first call. All sites down. Over 120 sites to be exact. This was not a normal outage, this must have been a change or something surely? What would take down such a network?

Now, maybe the configuration that was deployed was standard (actual Cisco configs from the configuration guide…..profile id 1234 lol) maybe, we had an underlying carrier issue? Nope. Sites were actually dropping in and out and getting the customer to reboot the router brought it back online…only for a few minutes.

The call came to me, I was the ‘one’ who had the best knowledge of GetVPN in the team. My colleague had worked out it was a GetVPN problem, an encryption problem. Yikes! That sounded technical and difficult. I spent many hours reading and learning about GetVPN when it was deployed at our workplace, but I still was no expert. See this technology relies on two very special routers, known as key servers. These guys are the backbone of the network, coordinating encryption keys to be handed out to every node. Depending on a configured time, they refresh and everyone is using the same key. If you have some nodes using one key and another using an older key, guess what happens? It’s like a Chinese person talking to Indian person, they can’t understand each others language.

We tried many things that morning, while I laid in bed on the phone, we rebooted key servers, got the customers to reboot any routers they could. Still nothing. That was about 5:30am. That was the time I decided we have to go into the office before everyone in the IT office gets there.

I got in the car and headed straight for work.

I was frantic in the car, still on the phone trying to work out what the hell happened!

We needed more information, we needed data to troubleshoot. Although we could not access the routers remotely to gather this data.

Arriving at work, it hit us. We have a lab. A sweet sweet lab and it was in our office. We could troubleshoot from here! We spent about 30 mins trying to debug and find the cause of the issue, but by this time it just all started to look the same. Managers started to come in, and demanded answers. They were not harsh, they were helpful but the entire WAN was down. We had to give constant updates…we are with TAC. Sites had to go to manual processes with absolutely no connection to the Data Centres. Phones didn’t work. No email no nothing. Imagine sitting at your house and you phone and internet was down and how annoyed you would be. Now multiple it by 120 sites and maybe at least 10 people per small site and 200 + at 5 large sites across the country. Is the correct answer C yet?

The next step after gathering all the logs, was Cisco TAC. This is technical support, from Cisco themselves. The experts.

I made the call and we got a guy from Texas. He was a GetVPN expert.

He was able to connect to my PC via Webex and found our first problem. Encryption was broken and when this happens you need to make sure that certain protocols in GetVPN are not encrypted, ever. This is so you can build the underlying connectivity using a routing protocol and also in case of an encryption problem you can still manage the routers.

Routing updates, ping and SSH should not be encrypted. SSH is already encrypted anyway. We modified this on the key servers and suddenly we had SSH access to all routers. More troubleshooting continued.

He found the problem, thank god.

Colleague performed a change on all remote routers a few days before. It was to update SSH keys for remote routers. Although by accident it included the key servers. It took the current generated keys used for both SSH and GetVPN encryption and removed them. During the morning connectivity was lost between the two key servers and they both became master. Then connectivity restored, but it was too late. The remote routers were still using the old keys from one key server and connectivity was lost. (as you can imagine it was even more technical than this, but this is all I remember).

Did you ask about Change Control? Yeah it was followed for the SSH key generation, but to be honest even with all my reading I still knew jack shit about GetVPN. The only way I really learned was when it broke.

So….make sure you lab things. Make sure you get your hands dirty, even if it is a virtual lab. It is the only way you will learn anything in life to be honest. Don’t be afraid to break things in the lab, watch what it does when it breaks and what it does when you restore it.

Don’t ever be afraid to ask for help, you will always learn something. Don’t ever give up either, if it has been broken then it can be fixed and don’t memorize the answers A,B,C & D because the question hasn’t been written yet in the real world!

So, the correct answer was not C, not all of the above or even phone a friend! It was when you deep down in the shit, escalate and ask for help. No one can be an expert at everything!

~Brad.

Network Engineer Interview

Recently I have been struggling with my career choices. I am a little confused about what I really want to do for the rest of my working life. Originally my first goal was just to wear a Cisco T-shirt to work! I now do that as often as I can to remind myself of my first goal.

My second goal was to be a ‘Core Network Engineer or Network Design Engineer’. I am currently working towards those now.

My brothers friend of mine recently contacted me to do an interview for her students at a TAFE in Victoria. I was more than happy to reply and help out upcoming students. This made me think back to when I was in TAFE. My teachers were great and really pushed us to get the best score possible. I was even asked by my teachers to apply for a Network Scholarship, although it wasn’t meant to be.

I did pass with pretty good marks at the Cisco Networking Academy and even though my worklife has been completely different than TAFE I have learnt some valuable lessons. Maybe one day, I could be a Cisco Networking Academy teacher?

Below is the interview I did for the friend. I hope it does help the students in some way.

  1. What kind of qualifications do you have?
  • Advanced Diploma of Computer Systems engineering
  • CCNA
  • CCDA
  • CCNP
  • CCDP
  1. How many years have you been in this industry?
  • I started in 2006 part time while still at TAFE, so about 9 years.
  1. What does a day in your job look like?
  • My current job is network support for mining companies. Typical day, if not on call is arriving at work and checking my sites I have been assigned. We use some monitoring systems which I can quickly check to see if any links or devices are down. Once that is complete I have a few projects I am working on, so I may have to check on them. Day to day I would be either making changes to switches or wireless infrastructure. Although to perform changes, we need to follow a change control procedure. We can only make on the fly changes if it is an emergency.

I usually have to speak with IS managers on the mine sites to get approvals to perform my changes. The work comes in via incidents (service desk) or by my manager. It could be a new VLAN to be pushed across the wireless network, a IPSec tunnel from our office to other remote offices or configuring an autonomous AP for wireless access in a piece of mining equipment. I might also receive small requests to update access control lists or adding new subnets to routers. Some bigger projects may be taking over management of existing switches and cleaning up configurations.

Usually when a major outage happens we need to do the troubleshooting, check power and comms to WAN router then go from there. Most incidents are related to power in the mining industry or unauthorized changes.

  1. What’s the worst network attack that you’ve ever seen?
  • I worked for a small ISP a year ago. We had a few different Internet pipes and we supplied either a secure internet (firewall in front of customer) or a non-secure (direct pipe to the internet). The customer would then need to provide own firewall. One night I was on call and started to receive some alerts from multiple customers. The firewall was being attacked, millions of half open TCP sessions where being created within the firewall and the firewall could no process the information. This caused the firewall to drop its routing neighborship with our PE router (Provider Edge) and took out every customers internet for about an hour. Manual intervention was required to black hole the traffic, direct it to Null0 (destination that doesn’t exist) which stopped the traffic flow.  The final fix, was to install a IDS (intrusion detection system) to automatically detect this type of attack and block the traffic before it made its way to firewall. One other attack I saw is Crypto-locker. Comes in an email and encrypts peoples hard drives so they cannot open it without paying a ransom to the hackers themselves to unlock.

If you want to see people trying to hack networks right now, the Norse corporation has deployed Honeypot servers (devices that look legitimate but are not) and they monitor attacks on this website –http://map.norsecorp.com/

  1. How much do you get paid per year?
  • 9 years ago when I started full time I got $25 dollars an hour while still at TAFE. I didn’t have my CCNA or any experience. As a contractor you could make $50 – $80 an hour as a CCNA with a couple of years’ experience back then as well. These days contracting pays the best, but there is no guarantee or work stability. Check out the current Hays Salary Centre for current rates depending on years’ experience. It also depends on the work you will be doing. Design & consulting pay the best.
  1. How do you set up a physical hardware firewall device on a network?
  • This really depends on the network design itself. Best practice is to have dual firewalls. You can deploy them in the active/passive configuration which means one is doing the work and the other is sitting there ready to take over. The active/active configuration is where traffic is shared or load balanced across the hardware. This is also dependent on the hardware itself as it needs to be able to support this configuration.
  1. What is the most difficult task that you have dealt with?
  • I had to install and troubleshoot a new wireless network. Unfortunately there was no wireless site survey done, they just installed Wireless AP’s where they thought it would be best. It took me a long time to try and get the network running and stable. They also had voice over the wireless which was not taken into consideration as well. Most enterprise wireless networks are controlled by a centralised device that can automatically change power and channels. Although if the RF environment has not been mapped out correctly, you are going to run into problems. Interference, rouge wireless AP’s and incorrect settings all played havoc with this network. I recommend always following the best practice guide lines when deploying a wireless network.
  1. If a client reports a wireless dead zone in their building what do you do to find the extent of it and then how would you fix it?
  • If the wireless is being monitored, a check of AP’s in the area and if they are all online is the first step. The next step would be to go to the area affected and use a wireless scanner (network Stumbler) to measure the signal strength of the AP in the dead spot. This will determine if the power level on the AP is high enough to service the area or with visual inspection you may find something that is causing interference. It could be the physical environment, a rouge AP or another wireless device. Depending on the wireless frequency you may have another AP in the area trying to use the same channel, a scan will pick this up.
  1. What is the most extreme problem you have come across?
  • Anything to do with entire WAN networks going down. Usually the WAN once deployed should be stable with redundancy in the design. The WAN is critical for major companies that access resources in the data centre. I was on call once and got a call at 4am in the morning. Half of our WAN started to fail, people could not access the data centre or internet, and out of 120 sites only half where working. It took 4 hours to fix this problem. It was caused by a change that previous night to all routers in the organization regarding SSH key generation and how the WAN communicated with encryption. We had to escalate to Cisco TAC (Cisco’s Technical Support Team) and an engineer from Texas found the issue and rectified. It was the worse outage I have ever had a phone call about.
  1. What is the most common problem you come across?
  • Problems like incorrect VLANs or duplex issues. People are in the wrong network, or the cabling is ruined causing major packet loss.
  1. How do you set up a wireless repeater/extender to get better signal in a room?
  • Ha! I have never actually used a wireless repeater! People use them in their homes, but enterprise access points are a lot more powerful regarding antennas and coverage. Usually in the industry it has been determined beforehand during the design process. If someone came up to me and asked me to deploy one, I would check the Cisco site and follow the instructions.

~Brad.