NOC Tier 2 Engineer

Role Summary

The Tier-2 NOC Engineer at Cubic Transportation Systems is responsible for taking escalations from Tier-1 support, restoring service across customer contracts spanning on-premises, Microsoft Azure, and AWS environments. This role involves troubleshooting across various layers including network, compute, operating system, database, application, and cloud. The engineer will engage vendors and internal teams, provide customer updates during major incidents, and contribute to the continuous improvement of the NOC by tuning alerts, automating runbooks, and raising problem records. Collaboration with the SOC is key for providing infrastructure data during security investigations.

Key Responsibilities

• Diagnose and Restore:
• Take escalated tickets from Tier-1 and troubleshoot across network, compute, OS, database, application, and cloud layers.
• Restore service through fixes, failover, scaling, or rolling back changes, adhering to contract SLAs.
• Identify and record the likely root cause with resolution notes in the ticket.
• Change Management:
• Raise and implement emergency changes required for service restoration.
• Roll back changes that cause incidents and verify if recent changes are the cause.
• Execute critical out-of-band patches requested by the SOC and validate service health post-patching.
• Coordination and Communication:
• Engage with vendors (Oracle, Microsoft, AWS, ISPs) and internal infrastructure and application teams.
• Provide status updates to customers and stakeholders during P1/P2 incidents.
• Collect and provide data (logs, configurations, flow data) to the SOC during security investigations.
• Continuous Improvement:
• Raise problem records for recurring or unexplained issues.
• Tune alerts by removing noisy ones, adjusting thresholds, and actioning false-positive requests from Tier-1.
• Write and improve runbooks, and automate common fixes to empower Tier-1.
• Coach Tier-1 analysts on troubleshooting techniques and runbook usage.

Scope and Escalation Boundaries

• Inputs this role works from:
• Escalated tickets from Tier-1 with triage evidence.
• Monitoring data from all tools, CMDB/service maps per contract, change records, and vendor contacts.
• Requests from SOC Tier-2 for infrastructure data.
• Escalate to NOC Tier-3 / Engineering when:
• The root cause is outside standard operations (e.g., code defect, architecture limit, product/IoT firmware, capacity design).
• A fix requires a significant change or design decision.
• The issue is not resolved within the P1/P2 time limit (e.g., 1 hour of active work on a P1).
• Escalate to the Incident Commander (war room) when:
• There is a major incident, affecting several contracts, a full customer outage, a likely SLA breach with penalties, or a potential security cause.

Requirements & Tech Stack

• Technical Depth: Hands-on depth in at least two of the following, with working knowledge of the others:
• Network: TCP/IP, OSI model, routing and switching, firewalls, VPN, DNS, DHCP, load balancers, network connectivity troubleshooting.
• Systems: Windows Server, Active Directory, Linux administration.
• Databases: Oracle and Microsoft SQL Server (performance, blocking, storage, job failures).
• Cloud: Microsoft Azure and AWS (compute, networking, identity, monitoring).
• Applications: Java-based applications, web and application servers, application logs.
• Working knowledge of products and IoT devices for supported contracts.
• Log analysis across firewalls, routers, switches, servers, and applications.
• Monitoring and Observability Tools:
• SolarWinds: NPM, NetFlow analysis, node and interface health.
• AppDynamics: transaction tracing and application performance analysis.
• Oracle Enterprise Manager: database and middleware performance analysis.
• Site24x7: synthetic monitoring and availability checks.
• Azure Monitor and AWS CloudWatch: metrics, alarms, and log queries.
• PRTG Network Monitor and Nagios.
• Ticketing systems (ServiceNow, Jira or similar) and Microsoft 365.
• Scripting Standards:
• Scripts stored in Git in an approved, peer-reviewed library.
• Read-only by default; diagnostic scripts do not alter systems. Scripts making changes require a change or ticket reference.
• No hard-coded credentials; credentials managed via a vault.
• Parameterized by contract and CI, safe to run multiple times, and logs actions to the ticket.
• Process Skills:
• ITIL incident, problem, and change management (including emergency changes and rollbacks).
• Vendor escalation and management.
• Shift handover and incident documentation.
• Professional Skills:
• Structured, analytical troubleshooting under pressure.
• Clear written and verbal technical updates for technical and customer audiences.
• Attention to detail and ability to manage multiple incidents concurrently.

Qualifications

• Preferred:
• Python for cross-system automation.
• Splunk SPL basics.
• Network device CLI for reading switch, router, and firewall state.
• Experience in a multi-customer or managed services NOC with contract-specific SLAs.
• Certifications (Preferred):
• ITIL 4 Foundation.
• CCNA or CompTIA Network+.
• Microsoft Certified: Azure Administrator Associate (AZ-104).
• AWS Certified SysOps Administrator – Associate.
• CompTIA Security+.