This is My Blog

2017/05/11
|

The journey to gigabit fibre with Octotel (and Vanilla)

A brief history

I was privileged enough to get onto the internet from home in 1995 courtesy of my parents returning from Hong Kong with a cutting edge 28.8k dial up modem. I’m not sure if that modem had different standards to what we used in South Africa, but it was a nightmare, it constantly disconnected without explanation. I believe this is what caused my quest for better internet to begin. Over the years I moved to a 33.6k leased line, one way satellite downlink (InfoSat) and to what I’ve been using for the last 10 years, ADSL, all while watching the rest of the world move ahead without us. I’ve never been satisfied as the bandwidth always lagged the bandwidth requirements of modern technologies, movies, cameras, software sizes, tools and so forth. Then came Octotel.
OctotelOctotel announced in 2015 that they would be laying fibre in my area, Sea Point, I was immediately thrilled. In 2016 I kept a close eye on the progress and the pricing started being released, with a gigabit option no less, the light was at the end of the tunnel. After attending a community meeting I was told my area would only be live in early 2017. 2017 rolled around and I could see the trenching progressing slowly towards my neighbourhood.

 

Choosing an ISP

Vanilla ISPI started scouting out the ISPs on Octotel. I definitely wanted gigabit, no compromises at this stage of the game. At the time there were only two ISPs offering gigabit on Octotel, one being Vanilla, I had never dealt with them before but they were responsive to my mails, I liked their philosophy as per their website and as I consider myself a ‘power user’ I think it was a good fit. Oh, and they support eSports teams (I enjoy the odd CS:GO tournament). Order placed. The total cost of the service would be R1668 including VAT (R969 for the line, R699 for uncapped bandwidth from Vanilla).

 

The process begins

The first step in the Octotel process is the trenching of the road and laying the conduits down in your street, once the trenching is done you basically have a network of empty pipes/conduits in your neighbourhood. My road was trenched roughly 5 weeks before my scheduled install date. As far as I can tell Octotel works in large areas. My road was at the end of the area they were working on at that stage so I would imagine trenching to install date could be anything from 5 weeks to 10 weeks depending on where you are in that area. Once trenched Octotel runs a link cable down the main roads and installs splitters in all the manholes linking the side streets. In my case this was done last, but should usually be done before your installation.

Octotel setting up manhole
Octotel setting up manhole splitter
My installation was scheduled for the 25th of April. A day before, my doorbell rang saying it was Octotel, a day early, I was in luck! Well, no, they needed me to move my motorbike so they could access the wall box in order to blow the fibre from my house to the main link on the main road, to be ready for the installers the following day. Blowing fibre is the process or taking a fibre cable (with some sort of plug I assume) and using high pressure air to literally blow the cable through the conduit until it comes out the other side. I managed to take a small video showing the blowing in action (its not that exciting).

 

Octotel blowing fibre
 

The next day, install day, the Octotel installation team came to splice up the fibre. I had prepared the point of entry, UPS and so on for the Fibre gear so all in all the install took about 1 hour to get the fibre from the wall box to the location of my router.

Octotel splicing
 

Once you’re spliced up and installed you’ll have what is known as an ONT (optical network terminal) on your premises connected to the GPON network.

Octotel ONT
Octotel ONT
 

As you can tell by the glaring red light, I was not connected at this stage. I consulted with Octotel and Vanilla and it turns out they hadn’t actually installed the main road fibre cable to the data center yet. I wasn’t thrilled to hear this of course.

In the following days of impatiently waiting to try and figure out when I would be live I tried to read more about the optical network, how it works, how it connects and so forth.

 

The Octotel GPON network

Disclaimer: Anything in this paragraph is inferred by material I have available to me, corrections welcome.

Octotel is not an ISP, all they do is provide the last mile connectivity from their data center where ISPs can peer with their infrastructure. They are known as an ‘Open Access Network’ which means they don’t compete with their own ISPs like Telkom/SAIX and Telkom Internet which can blur the lines and lead to unfair competition against ISPs.

They make use of gigabit passive optical network (GPON) technology to connect customers. This is a preferred technology for FTTx deployments because of the ability to connect multiple premises off a single fibre using splitters, the splitters are passive and do not require power so can be stored in places like manholes without cooling, power, etc.

Octotel Network Layout
Octotel Network layout
 

I’m not sure how many houses Octotel would put on a single fibre or splitter but as I understand it, the limit is 64. GPON is limited to 2.488 gigabit per second downlink and 1.244 gigabit per second on the uplink side, so if you had a congested network, contention would come into play, if all users were using their maximum allocation you would only receive 38 megabit per second down, 19 megabit per second up.

My best guess is that Octotel runs a Layer 2 network service for their ISPs, which basically means that when you plug your LAN cable into the ONT device, you are plugging in a network cable directly into your ISP network switch, just bridged by fibre.

 

Going Live, Speed Tests, and more

Unfortunately going live took quite a bit longer than anyone anticipated due to the missing cable in the main road, I was eventually live 17 days after my install date, I believe this is close to a ‘worst case’ as I was the first person installed in my area/zone as far as I know.

Having an internet connection faster that was actually as fast as my hardware could handle was a new paradigm for me. After some research I found the router I had (EdgeRouter X / ER-X) couldn’t actually do network address translation (NAT) at full gigabit wire speeds, only around 350 megabits per second, not good enough 🙂   I was recommended by Vanilla to try the Mikrotik RB3011UiAS-RM and after reading online this device seemed to be capable of handling the speed.

I used iperf3 on my network to determine what the best case switching speed was on my home LAN. I tested multiple switches and the lowest the throughput was 920 megabits per second with consistent sub 0.5ms local equipment latency. This is useful so we’ll know if we are hitting the limits of my own hardware.

The speed tests are more to illustrate the speed of Vanilla’s network and not Octotel, so keep that in mind when interpreting the results. Your ISP using Octotel may have different routing and performance which could produce better or worse results.

 

Basic Speedtests

In order to eliminate any possible delay my router would have on the connection I connected directly to the GPON ONT for these tests via LAN, so no switch, firewall etc.

For the following tests I made use of the speedtest-cli tool which is essentially the same as the in browser tool but produces information in an easier to digest format. Each result is based on an average of three tests.

Server Location Ping Download (Mb/s) Upload (Mb/s)
Vanilla Cape Town 3 898.6 25.74
Vodacom Johannesburg 22 189.09 25.20
London Web London 154 124.3 20.14

 

The test to Vanilla is essentially the highest achievable speed on this line as it is almost directly connected, doing a trace to the server I found only 3 hops in between me and the Vanilla speed test server. We can see here that the GPON network is basically as fast as my local gigabit LAN.  The speeds only dropped roughly 20 megabits per second from the iperf3 tests. The fastest test was only 5 megabits per second slower than local LAN.

 

It is clear to me that my router, even though it is a high quality router, it still can’t cope with full gigabit speeds if you want to use all its features. After moving the ONT from direct LAN to the router the top download speed remained basically the same, however, in order to do QoS on a Mikrotik router you need to disable the hardware NAT acceleration feature known as fasttrack. This dropped my speed down to around 650 megabits per second. I did some digging and could see that on a single connection test only one core of the router’s CPU is in use (of 2 cores), when doing a parallel connection test both cores get used and the performance then gets back up close to wire speeds. This single connection ‘slowdown’ is caused by my router’s CPU unable to process the firewall and QoS rules fast enough, not the connection itself. It’s nice to have these types of problems now 🙂

 

Latency tests

The basic throughput tests are a bit synthetic and may not really show the true experience a user has on the internet. One of my personal priorities has always been the latency of a connection and how it copes when under load. On ADSL people often assume lag or high latency spikes are caused by downloads but more often than not it is saturating the upload bandwidth that causes lag spikes and delays. This is the worst issue for gamers and people operating on remote servers requiring quick feedback.

I made use of the DSLReports speed test tool to test my line and lag.

The initial results were a bit of a surprise, I thought at 25 megabit per second upload buffer bloat would be a thing of the past but it turns out even this can be saturated and cause lag, here was my initial result.

Speedtest result no queue
As you can see the bufferbloat score is a D, which means when the upload line is saturated the latency will be increased substantially (lag). This isn’t the fault of the ISP or Octotel, merely the way my router handles the upload traffic.

Coming from ADSL, I have had to solve this problem before and am relatively experienced in doing so, which basically just involves configuring a QoS queue on my router and specify the maximum upload link speed and queuing algorithm of choice (I used PCQ in this case). Here is the next result with QoS enabled.

Speedtest result with PCQ enabled
Once the QoS was enabled controlling the upload speed of the router you can see everything looked brilliant.

 

Gaming tests

These tests are not very scientific I must warn you. I fired up Counter Strike Global Offensive (CS:GO) and played a few games. When matched on Cape Town servers my reported ping was between 3 and 8 milliseconds and when on Johannesburg servers between 22 and 30. I felt that the responsiveness (eg when a shot was registered by the server) was basically like a LAN game, where as on ADSL previously it had a minor delay before registering a shot.

In normal households you can’t stop your family from using the internet while you game. This can cause some lag and then family feuds 🙂    I started up some downloads and uploads while playing to see what the impact on latency was. In short, I had no noticeable increase in my ping at any stage.

I also downloaded a game from Steam just to test that aspect. I downloaded the older Counter Strike: Source which is 3.4 GB and it was completed in roughly 3 minutes and 20 seconds!

 

Streaming video tests

I don’t have a 4K TV at home so my tests were limited to 1080p on my streaming devices.

To keep it short and sweet, all video on Netflix, Amazon Prime Video and YouTube all loaded almost instantaneously and quickly upgraded themselves to 1080p video, even 1080p @ 60 fps in some cases where it was available. I never had an issue with mid-video buffering.

To test this a bit better I moved to my PC which has a near 4K display and tested some 4K streaming. This tested the connection a bit more but it was still able to play the video without any buffering which was great.

DSTV Catch up plus also had no problems and a 1.8GB HD movie took about 3 minutes to download.

 

Some thoughts

While my time with fibre is only just beginning, so far I’m extremely pleased with the results. Things like uptime and contention may appear later which I will be keeping an eye on and monitoring using some automated tools.

After using it for some time I rarely see a download (drivers, ISOs, updates, etc) go above 20 megabytes per second. It is still extremely fast but nowhere near saturating the gigabit capacity of the line or even the 650 megabit limit of my router. I would presume this is because most servers, peering links and so forth don’t really cater for these speeds as the norm.

I do however often saturate the upload speed of 25 megabits per second uploading application archives (Java WAR files) and videos so I may adjust my service to cater for this. I’d like to see Octotel offering a 1000 / 200 package as they offer a 200 / 200 package. Given than the upload and download channels on a GPON network are separate this should be easily possible. Even better would be a service similar to CyberSmart 400Mbps service, allowing users to change their speeds on the fly 500/500, 700/300, etc 🙂   wishful thinking perhaps 🙂

 

Conclusion

It seems my long, 22 year+, journey to a proper internet connection is finally coming to an end. Thanks Octotel and Vanilla for getting me across the line. When the XGPON upgrade comes, I’ll be signing up I’m sure 🙂

I hope this post has provided you with some useful information, if you have any questions about this post please comment or contact me and I will try and answer them for you.


7 Comments

  1. Alan Levin

    2017/05/15

    Donald, Thank you so much for this. Really well written article about your experience. I am hoping that you will allow us to demonstrate our real value by doing a few more tests to London, we do have over 600Mbps at any time and up to the full gigabit in quiet times. We find the Perpignan old Linux server can perform well at times... we also have at least 500Mbps to Joburg which is currently being upgraded to ensure at least the full gigabit is available...

  2. Donald Jackson

    2017/05/15

    Hi Alan, thanks for reaching out. I am aware the testing methodology isn't ideal, as there are so many factors influencing the results along the way. The international tests varied anywhere from 50 mbps all the way up to 200 mbps as you can see in the DSLReports tests. I'm going to write a follow up in the months to come after I've had some time with the connection so I can comment on the uptime and contention aspects and will certainly do more tests, probably with iperf and controlled servers next time just to eliminate some outside factors.

  3. Aaron

    2017/05/30

    Hi Donald

    Awesome review, I'm looking at this 1Gbps package as well, but still deciding whether or not I'll use even close to its potential.

    I would like to know, in regards to the router that Vanilla recommended, did you pay extra or was it supplied with the package?

    What is your avg download speed per second? 50mb/s? Or less?

  4. Donald Jackson

    2017/05/31

    Hi Aaron,

    The router recommendation for gigabit wasn't included in my case but if you opt to use the one provided I do believe that one is included (I speak under correction). Not satisfied originally with the 650 megabit/second I actually assembled a pfSense device with Intel NICs to do the full gigabit routing with QoS.

    After using the connection for about 3 weeks I rarely went above 200 megabit/sec (25 MB/sec) and according to my traffic graphs over a five minute period the highest I've sustained is 120 megabit/sec (thats 4500 megabytes in 5 minutes,download was probably complete quickly). I don't think this is the fault of Vanilla or Octotel, I just don't think the majority of servers in SA and otherwise cater for these kind of client speeds.

    Armed with these info/experience I decided to rather move to the 200 / 200 megabit package offered as I would have more use for higher upload speeds regularly than the odd high burst download speed.

    If they had a 1000/200 (or something better than 25 megabit/sec up) I would switch back to gigabit though, one can never have enough, and besides I have all the router gear to handle it now :)

    If you have any other questions I'll be happy to help.

  5. Aaron

    2017/05/31

    Hi Donald,

    Thanks for the view and reply, now I have a much clearer oath of what to choose.

    And you're right, one cannot have enough, but there's little to no use if you can't even use 80% of the lines potential, find it a waste.

    Please let us know how 200/200 works out, so far I find Vanilla other packages a bit more on the expensive side, only find the 1000/25 more bang for bucks. But will be interesting to know, since majority of ISP only offer 100/100...

    Thanks again
    Aaron

  6. Donald Jackson

    2017/05/31

    So far so good on the 200/200. I haven't found other ISPs offering the this service yet on Octotel but I must say the service from Vanilla has been pretty amazing so I think I'll stick with them for some time :)

  7. Alan Levin

    2017/06/02

    At Vanilla we have to charge a little more because we ensure that there is always over 500Mbps available to use in any direction (international or peering in Joburg or Cape Town). So in order to have the best performance and the best support all the way to your device we charge a little more. You can call us a boutique premium ISP :)

    We also very straight and I must admit that we experienced some serious disturbance on our network last night. It was the first time in 9 months although we take it very seriously and have already ordered redundant connections to ensure it can never happen again.


Would you like to share your thoughts?

Would you like to share your thoughts?

Leave a Reply