Software Engineer Resilience

About Datadog:

We're on a mission to build the best platform in the world for engineers to understand and scale their systems, applications, and teams. We operate at high scale-trillions of data points per day-providing always-on alerting, metrics visualization, logs, and application tracing for tens of thousands of companies. Our engineering culture values pragmatism, honesty, and simplicity to solve hard problems the right way.

The team:

The Resilience Engineering group at Datadog focuses on improving resilience in our software and staff. We work on defining our on-call tooling and incident response process for the entire company, constantly iterating on it through the lessons we learn from production. We help out during the most complex production incidents - our Resilience Engineers excel in troubleshooting and have a passion for problem solving and efficiency. We also build the chaos platform and tooling so that engineers can use a measured approach to break and test for system resilience and reproduce past bugs/incidents to verify their remediation.

The opportunity:

When we design systems, our Software Engineers and Site Reliability Engineers invest heavily on making them reliable and robust. However, it wouldn't be pragmatic to expect our systems to be perfect and never fail. Being prepared to deal with unknown failures both from a technical and organizational standpoint is the core work of Resilience Engineers.

You will:

• You will help with your expertise analyze complex issues in production and write postmortems in partnership with other engineering teams, but also help reproduce some of our past incidents in our staging and production environment at large scales to verify in practice our fixes (several teams, high complexity).
• You will get to contribute in the development of our self-service chaos platform implemented on top of Kubernetes.
• You will get to help define for the whole company how we respond to incidents and build tooling along the way to streamline that process. You will also help train our on-call staff, preparing newcomers to their on-call responsibilities but also refreshing the rest of the staff with what we've learnt from past incidents.

Requirements:

• You have significant programming experience and have a willingness to dive into unfamiliar codebases and find obscure bugs.
• You have architected, built, and operated distributed systems to solve problems at high scale in cloud-based environments.
• You have been on-call for critical systems and you have experience handling incidents using a formal organization process.
• You want to work in a fast-paced, high-growth environment that respects its engineers and customers.

Bonus points:

• You've worked on chaos engineering projects before.
• You've been an Incident Commander or have contributed to defining an incident response process.
• You have Linux/Kubernetes experience.
• You have experience in cross-team, cross-functional projects.