Wednesday, February 12, 2014

The chronicle of success

I want to start with a simple but important statement [and i hope i am not sued for breaking the NDA]: If there is one thing i enjoyed from the lab [how can someone enjoy a CCIE lab?], that is the appearance of IPv6 in so many tasks! It's refreshing to see it being used in so many imaginative ways.

Anyway, this is my complete story...

24/Oct/2013 - The experiment
It all started on 24th of October 2013 as an experiment. Some days later i was looking for excuses and i managed to find three of those. So i created my own CCIE SP Checklist and i decided to keep track of my progress through it.

04/Nov/2013 - Written exam
On 4th of November 2013 i took the written exam and i passed with one week of studying. Then i decided to take a one month break, during which i would deal with the projects at work that would run the following months in order to free some time later on, prepare the lab environment at home, book the lab exam, and most importantly create a schedule to follow until the lab exam. At the same time i would need to fix various open matters.

12/Dec/2013 - Lab study starts
This is the day i officially started my lab preparation and i added a poll just to get the feeling from people about my upcoming challenge. I also created a countdown/readiness gadget on the right, which would help me keep track of the deadline and my progress towards it. At the same time i created a page with what i consider the trio of success. The target was set from the beginning: pass. My initial readiness score was 46,5%.

15/Dec/2013 - Lab week #1
This was actually a three days week. Due to matters at work, i didn't manage to start at the scheduled day. Nevertheless, after an exhausting 11 hour lab, my readiness score increased up to 48,9%.

22/Dec/2013 - Lab week #2
This was the first full week spent on studying. Although my readiness score went up only to 50,6%, i spent a lot of time on GNS3 configurations.

29/Dec/2013 - Lab week #3
I used the Christmas period as an advantage of having extra days off from work for studying. My readiness score saw a major increase up to 54,2%.

05/Jan/2014 - Lab week #4
This was the most intense week of my studying. I used every possible time available during holidays to speed up my progress and i managed to increase my readiness score to 60,6%.

12/Jan/2014 - Lab week #5
This week i finished my GNS3 labs and i started my rack rentals sessions. Readiness score increased up to 64,5%.

19/Jan/2014 - Lab week #6
This week i started my PEC Gold Labs and i spent a lot of time on reading about areas that i felt i needed to improve my knowledge. My readiness score increased slightly up to 67,8%.

26/Jan/2014 - Lab week #7
I finished all full scale labs on rack rentals and all scheduled Gold Lab sessions, making my readiness score increase up to 70,3%.

02/Feb/2014 - Lab week #8
This was a week that i used to focus on core topics that i needed to gain more practice. As a result my readiness score increased up to 74,0%.

07/Feb/2014 - Lab week #9
This was the last week of my lab preparation. I slowed down my daily study and i focused on topics that i needed to boost in order to get that 60% score. The last two days i updated (aesthetically) all my NTS pages, i created a pdf from all of these and i recorded the critical ones in my mp3 player.

08/Feb/2014 - Flight to Brussels
In the morning i took the flight to Brussels. I took my mp3 player with me and two books that i found the most useful during my preparation. I arrived at the hotel just for lunch and then i had a visit at Cisco offices and the surrounding area. I spent the evening hours mainly doing a review of my NTS.

09/Feb/2014 - Review day
I spent the whole day in reviewing and listening to my notes, with various GNS3 breaks in between just to get my head cleared. My readiness score reached 77,3%, which is the one i went with to take the lab exam.

10/Feb/2014 - Lab exam

Wake up time. I have a quick shower and then i go downstairs and take my breakfast.

Time to leave the hotel and visit the Cisco offices. Outside Cisco another 10 people are also waiting for the door to open. After 5 minutes we all enter the building, we get our guest stickers and we wait for the proctor.

The proctor arrives (running late) and guides us to the exam room. He gives some introductory instructions and then we sign for our presence and we get our seat.

08:22 - Lab starts
I login and the first picture of the topology is shocking! I think i am going to have a hard time with so many devices. Then i click on the instructions and i am double shocked! The list of tasks seems endless. I start reading and taking notes on the scrap paper. Initial instructions say to verify various things before doing anything else, but as i do so, i notice some things that don't agree. Then looking at the topology i realize that this is probably going to be fixed later on by some other tasks, so i ignore it [first risk that proved correct]. While i keep writing down notes based on the instructions, i feel that this will take for ever. Time is close to 9:00 and i am still taking notes, while opening putty connections to devices, applying some config defaults and fixing the annoying putty defaults [whoever though of that putty defaults must be a psycho!]. At 9:20 i decide to stop reading all the details on the instructions; just write down the task subject, the technologies involved and the task points. While connecting to the IOS-XR boxes, i start getting an annoying (MPP related) message, the same one every minute on all of them. I inform the proctor of this, but he insists that this should be considered as part of the exam [sure, if you give me admin access then i could probably fix it]. So i have to choose: either stop the console messages of a specific priority or live with that message till the end of the exam [i didn't even think of filtering it at that time]. I decide to let it continue, since i need to keep an eye on every log produced in realtime to speed up my verification while configuration happens. I also open a notepad window which i will use for creating the IPv4/IPv6 TCL scripts and for creating/copying/pasting the configurations [as it proved out later, the second usage ended up being minimal].

I have started the first tasks, which can be considered as the silliest [when can a task be silly?] ones ever met. I couldn't find any real purpose of having them in the lab (especially in the form i met them), besides of somehow spending the candidate's time and making it easier to fail from the start of the exam. The initial configuration of all the devices is quite full, which might make things easier for some people and harder for others [including me]. Then i start moving on to the core topics. I draw a quick diagram of the complete core topology and i use that as a base for my protocol interaction mappings which are to follow on next tasks. Now, this is getting more interesting [at last something good about the exam]. Nothing difficult here, besides many lines of configurations on many devices on each task. Copy-paste doesn't seem very helpful in many tasks, because IOS & IOS-XR have different configuration formats. Also many tasks are worded in such a way that copy-paste wouldn't really help you a lot (it's like they made the exam in such a way that you need to type different things on many different devices). A later task requires clarification from the proctor, but again his answer isn't helpful at all [and he keeps on smiling]. I proceed with my own understanding of the requirements, which although longer in completion time, it's safer in results. For every task i configure, i have enabled all possible loggings in order to get realtime verification (at least from control-plane perspective). That way i am skipping data-plane verification from some devices, especially in tasks that TCL scripts cannot help. If i get the control-plane verified (from logging plus a single cli command) in 2/3 of the devices including at least half the IOS-XR boxes, then i can proceed with the next task. After all, if the tasks are known to me, i don't expect any surprises. While i think i'm moving with a good pace, the proctor's voice arrives from the far left to tear down that thought [and that was a shock].

11:50 - Lab break & lunch 
The proctor informs us 10 minutes before the break that we should all go for lunch and we should save our configurations before leaving the room. I have completed almost half of the tasks (<50 in points) and now i'm getting really nervous [and that's a rarely appearing feeling]. Based on my previous CCIE experience, i should have been way ahead (>60) in progress before the break for lunch. I stop completing tasks sequentially and i decide to immediately change my tactic. I quickly scan all the remaining tasks, i find the ones that seem the least dependent on others and require configuration mostly on non-core devices (based on the technologies/devices referred) and i decide to proceed with them after i return from the lunch. I count the points given by these tasks and if i don't make any mistakes, then i increase by a good amount my chances of passing. During the lunch [with limited mood and appetite] i'm thinking of other ways to speed up my progress too, especially in regards to task verification.

12:20 - Lunch ends & lab continues
I have already marked some tasks that can be completed independently of the others (assuming core functionality works fine, which it does), so i start working on them. Most of them are completed very fast, since they involved only [only?] four devices each. The last one although completed, its verification doesn't seem to work. I try some other variations, still no positive result. I decide to have a glimpse in the documentation for any hints that i may be missing, but i cannot find the documentation link (although i didn't actually spend more than 30 seconds looking for it [no way i was going to ask the proctor again]). I return to my initial configuration [the one i knew i should work] and then i try another method of verification (by the means of extra configuration). This time it works. At the same time the initial verification works too, even after i remove the extra configuration. Bug or feature? [what a surprise!] What matters is that now i am quite close to passing the lab and these "easy" tasks were a nice boost.

I am two hours before the lab ends and i have some more unresolved tasks to complete. I need to choose whether to verify the already completed tasks (and secure the existing points) or go for the unresolved ones (and get extra points, adding also the risk to lose points). I choose the second option [risky?], because i feel quite sure about most of my configurations (based on the initial verifications) and i want to get as many points as possible (to be honest i was also a little bit bored [how can someone be bored in a CCIE exam?] to try to verify again everything from the beginning). I feel like being on the edge with the current points and i need to increase them further. So i take all of the unresolved tasks from the beginning. There are three to four technologies involved in every task, with some very imaginative scenarios combining all of them in two point-critical tasks. All the tasks seem quite similar in terms of functionality to be implemented, and after examining thoroughly the provided topologies, i find an intriguing [almost diabolic] trap in the last one [damn you labels!]. This is the first task met that i hadn't practiced on during my two months preparation. This by itself is a challenge and an inner force is telling me to try it [ok, cut the crap obi wan]. But i don't have enough time left and there are other tasks that must be completed too. First priority is passing the exam, then accepting the challenge. So i focus on the tasks that i know well, leaving the challenge for the end.

Almost one hour before the end, i am finally starting to smile [that's the feeling i like]. I completed a few more tasks, did a quick semi-verification on them, IOS-XR doesn't work as expected, but i don't care. I know my solution is the correct one and the proctors can blame the IOS-XR software for its buginess [or do they expect me to open an SR?]. And now it's challenge time! [someone must be kidding] I already know what needs to be done in terms of separate technologies, so i just need to combine all the involved technologies, think of the gotchas and proceed to the configuration. 20 minutes later i have finished my initial configuration, but the verification isn't successful. That seems even trickier than i first though [well, i asked for it]. Second try and i change only a small part of the configuration. Now i am getting something towards the correct behavior, but still the verification isn't fully successful. Ok, now i am anxious again [but i'm smiling from the inside]. There are only 15 minutes left and this is the only task i haven't managed to tackle or even come near to a solution. But i'm not going to give up. Third try and a crazy idea comes to my mind [which is a little bit worrying]. I'll try a completely new approach. I just need to be sure that it won't affect my existing "underground" configuration. I bring the windows from all core routers in front of everything else, logging is always enabled (from the beginning of the exam) and i start configuring like a mad typist trying to break the world record [i must apologize to all other candidates if my typing these last minutes was annoying]. At 16:40 the proctor warns us that we must start saving the configurations because the lab ends in 12 minutes. I keep on typing and i complete my configuration some seconds after the proctor informs us that we should start exiting the lab environment. No time to verify it, copy/paste "wr" on every router and i click on "end lab" before the proctor starts giving warnings [and yellow cards].

16:52 - Lab ends
I feel quite sure about all my answers, except the one i didn't manage to verify. I just don't know if somehow later on (especially with the last task) i broke something that i had configured correctly in the beginning. This is a risk i took, but i don't know if i would do that again [no way i would do that again]. I didn't have time to double verify anything, so i trusted my initial semi-verification while completing each task. The lab structure is quite different from the R&S, because in most tasks you're asked to build blocks on top of other blocks and so on. So if something breaks in the initial tasks because of a change you made according to a latter task, you'll probably notice it from things not working in most of your tasks (assuming you have logging enabled for every possible technology); multicast might be an exception here. So besides two tasks that i didn't know whether my solution was according to the rules [thanks to proctor for giving vague answers], everything else was following strictly the lab instructions. Even if i was about to miss these two tasks (plus the one i couldn't verify at all, which was worth enough points), points collected from all other tasks would allow me to pass. Unless i missed completely something important and i didn't even notice it [or i messed everything up in the end].

I leave the Cisco building and i head to my hotel. Although it's now raining outside, i somehow don't feel annoyed by it [i didn't have an umbrella with me so the choices were limited]. Now i just have to wait for the ccie score email [and for my heart to calm down from the exam shock].

20:35 - Success
The much-awaited email arrives. That's probably a good sign, because it's been less than 4 hours after the end of my exam and i already got the results. Either everything went fine, or everything was screwed up; no middle choices in such a short time [this is something i invented while waiting]. I login to the CCIE portal and i see the PASS result (i still keep my CCIE page clear from fails, both in written and lab exams). Another amazing journey has come to an and. After half an hour looking at the result, i submit my lab critique describing my experience with this lab exam [and believe me it's wasn't nice].

So, what were the major differences from my previous CCIE, besides the minus one month of preparation?

First of all, the exam difficulty (in terms of technologies involved) was about the same, maybe a little bit harder; it was easier than what i was prepared for, but definitely not something i would consider as generally easy; i would give it an 8-8,5 out of 10.  The large difference was that this time the devices were more, the tasks were more, the configurations were more...but the time was exactly the same. I can't remember a single task that required configuration on one or even two devices and that with just a few lines on each.

This time i decided to keep my lab preparation notes on a public place; and what's better than the same blog i am publishing my progress? You can find all of them under the NTS page. By making public something of technical nature you feel obliged to the people that read you to provide useful and (hopefully) concise information. It may have cost me quite a few of hours doing it on a daily basis, but i don't regret it at all. It proved very useful to me too and it made the recording process much easier.

Also i decided to keep track of my study and progress in various xls files, as you can see in the exam progress page. This helped me continuously evaluate my knowledge and focus on specific things that i needed to improve. It was like a project that i needed to complete in a very specific time. Also it's always interesting to study graphs instead of numbers that alone might not say too much. I knew which were the core topics and i spent most of time on them; but i wanted to be sure that i won't miss anything non-core because i had no idea of how to use it. So 60% became my need-to-pass base on every topic, while core topics should range even higher, from 70% to 90%.

Regarding the lab preparation, as already described in my home/virtual labs page, since i couldn't find any interesting grading labs, i did the grading myself. And if i wanted to offer a hint for this, i would say that you have to try to be as honest and strict as possible with yourself. Believe me, it will pay you back. I mainly used INE's topology of full scale labs customized for GNS3 and the PEC Gold Labs for this type of preparation, while in my previous CCIE i had used INE's mock labs and Cisco's assessor labs.

Last but not least, while being in preparation, i configured almost every possible scenario and technology included in the CCIE blueprint in IOS and IOS-XR devices. I knew that IOS-XR was important in the CCIE SP lab and i wanted to test everything on that too. I didn't actually try everything in full, i.e. check the produced outputs and all this staff, but i did study and configure everything just to get an idea of what i should do if that topic was included in the exam. It's obvious that core topics were configured and tested as thoroughly as possible. And this is where the PEC Gold Labs helped me a lot. For the last 3 Gold Labs, i was able to finish my Gold Lab in a very short time, so for the rest of the lab time i used the IOS-XR equipment just to test my own scenarios. I did something similar with INE's rack rentals too, after i finished all full scale labs. Although i did have access to IOS-XR devices in my work, for personal reasons i didn't want to use them for testing things.

Before closing this long post, there is one question that comes to my mind the last few hours: Would i pass if i had studied for only one month? I will avoid giving a definite answer, but if i knew that configuration speed played such an important role from the beginning of my preparation, then i would have focused more on that and less on technology inners (which i find annoying if being promoted by Cisco).

What is my advice to everyone thinking of trying the current CCIE SP lab? Learn the core technologies (IGPs, MP-BGP, MPLS/TE, Multicast, IPv6) inside-out, combine and test them in all possible scenarios (one above the other, one combined with the other, multiple combinations above or below other technologies, etc.) and then focus on configuration speed. Think fast, act faster. When you become a master on that, spend some time on every other topic too. And don't risk any challenges, unless you're crazy like me!

Total lab cost: 2.663€

PS: Did i say i enjoyed IPv6 too much?

And some interesting numbers from my exam progress page:

Written study days: 7
Written study hours: 37 
Lab study days: 59
Lab study hours: 354,5
Total study days: 66
Total study hours: 391,5
Average lab study hours per day: 6 (from which ~1,5 hours on avg were spent daily on this blog)

My proposed hints for passing the CCIE SP lab:
  • Read the RFCs, participate in IETF WGs
  • Use GNS3 for your IOS (and soon IOS-XR) needs
  • Use INE's rack rentals or PEC Gold Labs or your work's lab for your IOS-XR needs
  • Use INE's full scale labs for testing and improving your readiness
  • Keep track of your progress, rate yourself as much as possible but always be honest
  • Enable IPv6 on something every day [and disable IPv4 on something every day]
  • Live with MPLS, sleep with MPLS, eat with MPLS, dream of MPLS [is that a female?]
  • Master the trio of success
  • Don't be afraid of changing early your lab tactic if it doesn't work for you
  • Take typing lessons [at least you should easily get another job after that]


  1. I have just one question. Did you draw the entire topology or did each section have a mini diagram relating to the task?

  2. Each major section had a mini diagram, but i mainly used the whole topology diagram (otherwise the screen was getting full of windows and i was trying to minimize their annoying effect). Also i was getting the idea of the screen becoming sluggish as i was opening more windows.
    My single drawing (core topology) was used solely for mapping protocol interactions on it.

    1. Thanks mate. I am almost there, I think. Based on your experience would you recommend drawing a diagram or just using the ones provided in the lab. I have read some blogs where drawing the entire topology has been suggested.

    2. The only advantage i can think of drawing the entire topology is to avoid looking it at your screen, going back and forth between it and the putty windows. If Cisco provided a 2nd monitor just for the topologies, then i don't think i would ever draw anything. But this is just me. On the other hand, maybe drawing the entire topology could earn me some time and make the lab easier for me. It's your choice. Spend some minutes exploring the environment and if it doesn't suit you, make the drawings.

      On my lab critique i entered a comment where i described my idea of a perfect lab environment: provide a 2nd monitor just for the topologies and the lab tasks. The windows of the topologies should allow someone to draw transparently some basic shapes and text upon them having the initial topology as a background...just like annotating in webex or any other conferencing tool.

  3. Congratulations and thanks for the awesome blog!!

    Would you mind to share the two books titles that you took along for reading? In addition you that any other books docs that you would recommend?


    1. The two books i took with me were "MPLS-Enabled Applications: Emerging Developments and New Technologies" and "MPLS Configuration on Cisco IOS Software".
      The first is one of the best i have read the last few years (i could call it the MPLS bible for network architects), but it gets quite difficult in some parts (it uses RFCs for reference, something that i don't expect any actual config examples into it).
      Regarding the second one, i don't think you'll find another book with so many Cisco configuration examples and diagrams about all MPLS-related technologies/topologies/scenarios.
      Of course both books have their shortcomings (the 2nd one unfortunately has quite a few of errors, so you need to know some things in advance in order to catch them), but in general they are a must for everyone dealing with advanced MPLS in his/her daily job.
      Other than these two, Ivan's "MPLS and VPN Architectures" vol 1 & 2 are excellent, plus Doyle's "Routing TCP/IP" volumes which always remain on top for anything ip related. Last but not least, "Interdomain Multicast Routing" is a must for any type of carrier and inter-provider multicast.

    2. Dear Tassos,
      Great story and very inspirational , I know its bit too late since I red your story but its great and thumbs up for your great success in little time,

      I have lab exam in few days,

      I have one concern after completing the lab do we have to save the running config,

      copy run start
      these are the commands or else,

      and what about XR saving running config.

  4. Hi Tassos

    Firstly what a great resource for CCIE-SP candidates this was undoubtedly one of the main sources of references behind doc cd for me.

    I watched a Cisco Live presentation on next generation mvpn, wondering if you've had a chance to play around with any? There is like 27 deployment options so it's quite intensive, here is the link

    With you're depth I'd love to see a similar blog on this subject