500-240 download - Cisco Mobile Backhaul for Field Engineers (CMBFE) Updated: 2023

Optus has submitted to the Senate Standing Committee on Environment and Communications a full account on what it thinks caused the massive outage last 8 November that crippled its subscribers: 90 Cisco provider edge routers automatically self-isolated to protect themselves from an overload of IP routing information.

Previously, the telco attributed the outage to a regular upgrade going wrong that fouled up its network.

It claimed that an "international peering network" had fed it bad data.

Last Monday, the company's spokesperson said: "At around 4:05am Wednesday morning, the Optus network received changes to routing information from an international peering network following a routine software upgrade."

"These routing information changes propagated through multiple layers in our network and exceeded preset safety levels on key routers which could not handle these. This resulted in those routers disconnecting from the Optus IP Core network to protect themselves."

According to news reports, the unnamed "international peering network" that contributed to the outage is run by its Singaporean parent company Singtel.

But Singel disputed the claims, pinpointing that the outage was caused by Optus' safety mechanisms, and not a routine software upgrade.

Optus confirmed Singtel's account in a submission filed late on Thursday, ahead of a senate appearance today.

The Optus report reads: "It is now understood that the outage occurred due to approximately 90 PE routers automatically self-isolating in order to protect themselves from an overload of IP routing information."

"These self-protection limits are default settings provided by the relevant global equipment vendor (Cisco)."

Optus said the unexpected overload of IP routing information occured after a "software upgrade at one of the Singtel internet exchange" in North America.

During the upgrade, Optus said its network received changes in routing information from an alternate Singtel peering router.

"These routing changes were propagated through multiple layers of our IP Core network."

Optus suggests the upgrade led to the bad route information being propagated. However, it did not explain further, but now said this "was not the cause of the incident in Australia."

One hundred fifty engineers and technicians are now working to restore the system, supported by another 250 staff and five vendors, according to the document.

For the first six hours of the outage, engineers tried to come up with explanations for the large-scale outage.

Theories include whether works overnight by Optus itself were the cause but it found no resolution.

It also explored the possibility of a DDoS attack, a network authentication issue, or problems with other vendors such as its content delivery network provider.

But its IPv6 line of enquiry became its "leading hypothesis for network restoration."

“Through this process, we identified that resetting routing connectivity addressed the loss of network services. This occurred at 10:21am."

Optus' engineers then performed the following steps:

1. Resetting and clearing routing connectivity on network elements which had disconnected themselves from the network.

2. Physically rebooting and reconnecting some network elements to restore connectivity.

3. Carefully and methodically re-introducing traffic onto the mobile data and voice core to avoid a signalling surge on the network.

Optus continued to investigate more on the matter, performing unspecified "resiliency" works on the network between 8 November to 13 November.

The submission also detailed its customer communications defence on the outage day, but will likely to be challenged in a senate inquiry.

Optus has offered a 200GB data to affected consumers, but some say this is not enough.

The data is not enough to compensate the affected operations of businesses which lost money during the outage. However, the telco argued that if there is an outage, businesses must be responsible to own a backup connectivity in the event service is down.

In its submission, Optus argued making a telco pay financial compensation for losses is not a precedent that should be set.

"However, there is no precedent for compensation being paid by telecommunications providers to all business customers who suffer a loss of business as a result of an outage of the kind that occurred on 8 November, either here or overseas."

It argued that there no precedent for essential services such as electricity providers to pay compensation for businesses losses when there is an outage.

Electricity networks do not compensate business customers for "consequential losses" such as wages. It cited Ausgrid, an NSW-based electricity provider, which stated: "There is no compensation granted for consequential loss such as wages, productivity or trade.”

Optus downplayed the outage, saying "it isn't the first to suffer a sizeable outage in Australia" nor would the 8 November outage be the last incident of its type.

"While every communications network provider wants to avoid such outcomes, it is an unfortunate reality in our reliant digital age that no communications network can completely protect against, nor prevent, these types of occurrences from ever happening – despite the investments made or resiliency efforts undertaken."

Source: Ken Wolter / Shutterstock.com

Source: Sasima / Shutterstock.com

Source: The Art of Pics / Shutterstock.com

Michael Que is a financial writer with extensive experience in the technology industry, with his work featured on Seeking Alpha, Benzinga and MSN Money. He is the owner of Que Capital, a research firm that combines fundamental analysis with ESG factors to pick the best sustainable long-term investments.

We’ve narrowed IT certifications into specific categories to help IT professionals assess what’s available and pursue the best certifications to show their willingness to learn and develop the in-demand career skills employers want.

Best database certifications 

Database platforms have changed greatly over the years, but database technology remains important for various applications and computing tasks. Available certifications for IT professionals include those for database administrators (DBAs), database developers, data analysts and architects, business intelligence, and data warehousing specialists, and other data professionals.

Obtaining database certifications demonstrates an understanding of database concepts, design, implementation, administration and security. This can boost your credibility in the job market and show potential employers that you have the skills needed to work with databases. The best database certifications include the following:

Best SAS certifications 

SAS is one of the world’s leading firms for business analytics, data warehousing and data mining. Today, the SAS Global Certification Program offers 23 credentials across categories including foundation tools, advanced analytics, business intelligence, data management and administration.

SAS programmers remain in high demand, with a quick search of job boards showing thousands of open positions. Obtaining SAS certification shows employers that you are proficient in the company’s popular suite of tools. Some of SAS’s certification programs include the following: 

Many professionals earn certifications to help navigate their career paths. According to the IT Salary Report, 92 percent of information technology professionals have at least one certification.

Best Cisco certifications 

Cisco Systems is a market leader not only in networking and communications products, but also storage networking and solutions for data centers. Cisco offers a variety of certifications for IT professionals, ranging from entry level credentials to expert-level exams. 

These certifications prepare professionals for Cisco-related careers. A search of job boards reveals thousands of open positions for Cisco experts, underscoring the continued relevance of these skills. Some of Cisco’s certifications include the following:

Best Dell certifications 

Dell Technologies remains one of the world’s leading computing companies. In addition to its well-known hardware lineup, Dell also offers solutions for networks, storage, servers, gateways and embedded computing, as well as a broad range of IT and business services.

Becoming certified in Dell products can help make IT professionals competitive in engineering roles for server, virtualization, networking, systems, integration and data security. Additional roles include consultants, account executives, system administrators, IT managers and deployment managers.

Best mobility certifications 

In the mobile era, it has become increasingly important for network engineers to support local, remote and mobile users, as well as provide proper infrastructure. The focus on application and app development now leans more toward mobile environments, requiring security professionals to thoroughly address mobility from all perspectives.

Due to the fast-changing nature of mobile technology, not many mobility certifications have become widely adopted. However, a few of the top mobility certifications can help IT professionals stand out in this rapidly evolving field. 

If part of your job includes selling and implementing an IT solution, you may want to pursue the best sales certifications. You’ll show your organization that you’re willing to go above and beyond to reach sales targets.

Best computer hardware certifications 

As remote and computer-based work has become more common, it’s more important than ever that businesses and individuals be able to maintain their hardware. While discussions about potential computer-related jobs often revolve around software work and coding, jumping into the IT field by becoming a computer technician is an excellent starting point.

Today, thousands of hardware technician jobs are available across the country. Entering this industry becomes more accessible for those who acquire computer hardware certifications. These certifications can showcase your expertise and proficiency in the upkeep of computers, mobile devices, printers and other hardware components.

Best Google Cloud certifications 

IT pros with solid cloud computing skills continue to be in high demand as more companies adopt cloud technologies. Today, Google Cloud is one of the market leaders in the cloud computing space. 

Regardless of where you are in your IT career, engaging with certification programs can demonstrate your willingness to keep on top of rapidly evolving cloud technologies. To that end, Google has introduced a host of certifications for its cloud platform, including the following: 

Best evergreen IT certifications

In the fast-changing world of technology, it can help to focus on certifications that have stood the test of time. “Evergreen” refers to certifications that remain popular year after year. 

The top evergreen certifications are based on accurate pay surveys in IT, reports from IT professionals about certifications they want or pursue the most, and those that appear most frequently in online job postings. Obtaining these credentials is one step toward ensuring that your skills remain relevant for a long time: 

Best IT governance certifications 

IT governance provides structure for aligning a company’s IT with its business strategies. Organizations faced with compliance rigors always need experienced IT pros who can see the big picture and understand technology risks. This means certified IT governance professionals are likely to remain in high demand.

Earning one of the following certifications proves a commitment to understanding the role of IT governance and its position in a company’s current and future success. Getting certified can validate your expert knowledge and lead to advanced career opportunities.

Best system administrator certifications 

An IT system administrator is responsible for managing and maintaining the information technology infrastructure within an organization. The position demands sought-after career skills, ranging from configuring and maintaining servers and clients to managing access controls, network services, and addressing application resource requirements.

If you’re in charge of managing modern servers, there’s a long list of tools and technologies that system administrators must master. Obtaining some of the most prominent system administrator certifications can demonstrate your mastery to potential employers. 

Best ITIL certifications 

ITIL, or Information Technology Infrastructure Library, was developed to establish standardized best practices for IT services within government agencies. Over the ensuing four decades, businesses of all types embraced, modified, and extended ITIL, shaping it into a comprehensive framework for managing IT service delivery. 

The ITIL framework remains the benchmark for best practices in IT service and delivery management, offering certification programs that cater to IT professionals at all levels. These training and certification courses ensure that IT professionals stay well-prepared for the ongoing evolution in IT service delivery management. There are four certifications in the ITIL certification program:

Best enterprise architect certifications 

An IT enterprise architect is responsible for designing and managing the overall structure and framework of an organization’s information technology system. Enterprise architect certifications are among the highest that an IT professional can achieve; fewer than 1 percent ultimately reach this level. 

Enterprise architects are among the highest-paid employees and consultants in the tech industry. These certifications can put IT professionals on a path to many lucrative positions. The average worker earns over six figures annually. Some top enterprise architect certifications are listed below:

To become an enterprise IT architect, you’ll need knowledge of systems deployment, design and architecture, as well as a strong business foundation.

Best CompTIA certifications

CompTIA is a nonprofit trade association made up of more than 2,000 member organizations and 3,000 business partners. The organization’s vendor-neutral certification program is one of the best recognized in the IT industry. Since CompTIA developed its A+ credential in 1993, it has issued more than two million certifications.

CompTIA certifications are grouped by skill set and focus on the real-world skills IT professionals need. Armed with these credentials, you can demonstrate that you know how to manage and support IT infrastructure. 

Best Oracle certifications 

A longtime leader in database software, Oracle also offers cloud solutions, servers, engineered systems, storage, and more. The company has more than 430,000 customers in 175 countries. 

Today, Oracle’s training program offers six certification levels that span 16 product categories with more than 200 individual credentials. Considering the depth and breadth of this program — and the number of Oracle customers — it’s no surprise that Oracle certifications are highly sought after. 

Vendor-specific certifications address a particular vendor’s hardware and software. For example, you can pursue Oracle certifications and Dell certifications to become an expert in those companies’ environments.

Best business continuity and disaster recovery certifications

Business continuity and disaster recovery keep systems running and data available in the event of interruptions or faults. These programs bring systems back to normal operation after a disaster has occurred.

Business continuity and disaster recovery certifications are seeing a healthy uptrend as new cloud-based tools grow in popularity. While business continuity planning and disaster recovery planning have always been essential, they’re becoming more critical than ever — and IT certifications are following suit.