AA228V/CS238V
Validation of Safety-Critical Systems
Lecture 01: Introduction
Sydney Katz (smkatz@stanford.edu)
Yay!
Algorithms for Validation
AA228V/CS238V
Algorithms for Optimization Algorithms for Decision Making Algorithms for Validation
AA222/CS361
AA228/CS238
AA228V/CS238V
Design Stuff
Check If It Works
Algorithms for Optimization Algorithms for Decision Making Algorithms for Validation
AA222/CS361
AA228/CS238
AA228V/CS238V
Find out / around chart
***
Algorithms for Optimization Algorithms for Decision Making
AA222/CS361
AA228/CS238
Algorithms for Validation
AA228V/CS238V
*In a controlled, safe environment such as a simulation instead of the real world
*In a controlled, safe environment such as a simulation instead of the real world
… unlike these people.
Side Note
Did you guys see the humanoid robot games this year?
System
Specification
Validation
Algorithm
Information
System
Specification
Validation
Algorithm
Information
Failure Analysis
System
Specification
Validation
Algorithm
Information
Failure Analysis
Formal Guarantees
System
Specification
Validation
Algorithm
Information
Failure Analysis
Formal Guarantees
Explanations
System
Specification
Validation
Algorithm
Information
Failure Analysis
Formal Guarantees
Explanations
Runtime Assurances
Failure Analysis
Formal Guarantees
Explanations
Runtime Assurances
There is no silver bullet in safety validation.
We must build a safety case.
This course is a good place to start!
Plan for Today
Plan for Today
1
Course Staff Intros
2
Convince You That Safety Validation Is Important
3
Introduce Validation Framework for the Course
4
Course Logistics
Course Staff Intros
Course Staff Intros
Sydney Katz
Instructor
TA Name
TA
TA Name
TA
Sydney Katz
Instructor
2014
2017
2018
2020
2023
2025
B.S. Electrical Engineering
MIT Lincoln Laboratory
M.S. Aeronautics
and Astronautics
PhD Aeronautics
and Astronautics
Postdoc
Algorithms for Validation
Co-founder
Sydney Katz
Instructor
I collect bouncy balls.
I never got my wisdom teeth.
I was born in the same hospital as LeBron James and Steph Curry.
I learned to ride a bike when I was 23.
I used to jump rope competitively.
Jump rope
Course Staff Intros
Sydney Katz
Instructor
TA Name
TA
TA Name
TA
Plan for Today
1
Course Staff Intros
2
Convince You That Safety Validation Is Important
3
Introduce Validation Framework for the Course
4
Course Logistics
Convince You That Safety Validation Is Important
Robots (and Michael Scott) will do exactly what we tell them to do
Robots (and Michael Scott) will do exactly what we tell them to do
… so we need to give them good instructions
Robots (and Michael Scott and some students) will do exactly what we tell them to do
… so we need to give them good instructions
Robots (and Michael Scott and some students) will do exactly what we tell them to do
… so we need to give them good instructions
Alignment Problem
Alignment Problem
Imperfect objective
Imperfect model
Imperfect optimization
Alignment Problem
Imperfect objective
Imperfect model
Imperfect optimization
Alignment Problem
Imperfect objective
Imperfect model
Imperfect optimization
Objective: maximize score
≠  win race
Credit: OpenAI
Alignment Problem
Imperfect objective
Imperfect model
Imperfect optimization
next token prediction  ≠  fair code
Alignment Problem
Imperfect objective
Imperfect model
Imperfect optimization
reward web-tool use  ≠  search when needed
OpenAI Developer CommunityJan 2026
What is oai-calculator and why is GPT calling it often?
"type": "web_search_call",
"query": "calculator: 1+1"
OpenAI Alignment2025
A training-time bug "inadvertently rewarded superficial web-tool use" in GPT-5.1, so it used the browser as a calculator while acting as if it had searched.
Sources: OpenAI Developer Community; alignment.openai.com/prod-evals
Alignment Problem
Imperfect objective
Imperfect model
Imperfect optimization
reward: solve ExploitGym  ≠  solve it legitimately
FortuneJul 21, 2026
OpenAI says its AI models escaped from a secure test environment and hacked into AI company Hugging Face in order to cheat on an evaluation
MIT Technology ReviewAug 26, 2026
The inside story on why OpenAI agents hacked Hugging Face
The underlying models had been rewarded for cheating and communicating with each other.
Sources: Fortune; MIT Technology Review
Alignment Problem
Imperfect objective
Imperfect model
Imperfect optimization
Long-Term Capital Management (LTCM) Hedge Fund
Credit: courses.cit.cornell.edu/engrwords/progress_reports/Baek_et_al_LTCM_PR.pdf
Alignment Problem
Imperfect objective
Imperfect model
Imperfect optimization
Long-Term Capital Management (LTCM) Hedge Fund
Alignment Problem
Imperfect objective
Imperfect model
Imperfect optimization
+1
Alignment Problem
Imperfect objective
Imperfect model
Imperfect optimization
The
Alignment Problem
motivates
the need for validation.
Especially for complex systems that operate in safety-critical settings.
Especially for complex systems that operate in safety-critical settings.
Therac-25 radiation machine
Credit: pvs-studio.com/en/blog/posts/0438/
Boeing 737 Max
Credit: Boeing
Autonomous vehicles
Credit: KTVU-TV/AP
Convinced that we need validation?
Convinced that we need validation?
Many companies and regulatory agencies are.
DO-178C Software Considerations in Airborne Systems and Equipment Certification
Credit: rapitasystems.com/do178
DO-178C Software Considerations in Airborne Systems and Equipment Certification
Credit: FAA
Credit: EASA
Waymo Safety Impact
271.3M rider-only miles
through June 2026 · 5 metro areas
82%
fewer injury-causing crashes than human drivers
95%
fewer serious-injury-or-worse crashes
93%
fewer injury crashes involving pedestrians
Source: waymo.com/safety/impact (Sept 2026)
Anthropic
Responsible Scaling Policy
v3.4 · effective July 8, 2026 · first version Sept 2023
Capability thresholds
Evaluate models against defined dangerous-capability thresholds before and during deployment.
Frontier Safety Roadmap
Published goals for improving risk mitigations over time.
Risk Reports
Quantify risk across all deployed models, with external expert review.
Internal red-teaming
Aim to find jailbreaks better than their public bug-bounty participants.
Source: anthropic.com/responsible-scaling-policy
And they are hiring!
WaymoNow hiring
Safety Engineer, Risk Management$204K–$252K
System Safety Engineersafety cases, STPA
Senior Formal Verification Engineer$204K–$259K
waymo.com/careers · 2026
OpenAINow hiring
Researcher, Safety Oversight$380K–$500K
Researcher, Alignment$295K–$500K
Red Team Specialist, Cyber$198K–$320K
openai.com/careers · Oct 2026
Google DeepMindNow hiring
Research Engineer, AGI Safety & Alignment
Research Scientist, Frontier Safety Framework
Research Engineer, Frontier Safety Mitigations
deepmind.google/careers · 2026
AnthropicNow hiring
Research Scientist, Interpretability$350K–$850K
Research Scientist, Biological Safety$300K–$405K
Security Risk Analyst, Risk Engineering$270K–$345K
anthropic.com/careers · Oct 2026
And they are hiring!
TeslaNow hiring
System Validation Engineer, Optimus Hand$92K–$317K
Software Validation Engineer, Self-Driving$113K–$303K
Safety & Restraints Validation Engineer$92K–$196K
tesla.com/careers · 2026
ZooxNow hiring
Sensing & Perception System Safety Engineer$170K–$267K
Principal Systems Engineer, Safety$313K–$376K
System Safety Intern$5.5K–$9.5K/mo
zoox.com/careers · 2026
Joby AviationNow hiring
System Safety Engineer$127K–$160K
Autonomous Aircraft Systems Eng. Manager$191K–$300K
Battery Safety Engineer$107K–$185K
jobyaviation.com/careers · 2026
Shield AINow hiring
Sr. Staff Verification & Validation Engineer$170K–$240K
Sr. Engineer, System Integration Test$110K–$170K
Staff Engineer, Systems$142K–$213K
shield.ai/careers · 2026
Convinced now?
Convinced now?
Hope so because we need to move on.
Plan for Today
1
Course Staff Intros
2
Convince You That Safety Validation Is Important
3
Introduce Validation Framework for the Course
Introduce Validation Framework for the Course
4
Course Logistics
Introduce Validation Framework for the Course
System
Specification
Validation
Algorithm
System
Specification
Validation
Algorithm
Ch 2. System Modeling
Ch 3. Property Specification
Ch 4. Falsification through Optimization
Ch 5. Falsification through Planning
Ch 6. Failure Distribution
Ch 7. Failure Probability Estimation
Ch 8. Reachability for Linear Systems
Ch 9. Reachability for Nonlinear Systems
Ch 10. Reachability for Discrete Systems
Ch 11. Explainability
Ch 12. Runtime Monitoring
System
Agent
Environment
Sensor
System
Agent
Environment
Sensor
System
Agent
Environment
Sensor
Policy
System
Agent
Environment
Sensor
Policy
Transition Model
System
Agent
Environment
Sensor
Policy
Transition Model
Observation Model
θ
ω
System
Agent
Environment
Sensor
Policy
Transition Model
Observation Model
=[,]
θ
ω
System
Agent
Environment
Sensor
Policy
Transition Model
Observation Model
=[,]
θ
ω
θˆ
ωˆ
System
Agent
Environment
Sensor
Policy
Transition Model
Observation Model
=[,]
θ
ω
=[,]
θˆ
ωˆ
System
Agent
Environment
Sensor
Policy
Transition Model
Observation Model
=[,]
θ
ω
=[,]
θˆ
ωˆ
System
Agent
Environment
Sensor
Policy
Transition Model
Observation Model
=[,]
θ
ω
=[,]
θˆ
ωˆ
System
Agent
Environment
Sensor
System
Agent
Environment
Sensor
System
Agent
Environment
Sensor
System
Agent
Environment
Sensor
System
Agent
Environment
Sensor
System
Agent
Environment
Sensor
System
Agent
Environment
Sensor
System
Agent
Environment
Sensor
Agent
Environment
Sensor
…
Agent
Environment
Sensor
System
Agent
Environment
Sensor
Agent
Environment
Sensor
…
Agent
Environment
Sensor
System
System
System
Specification
Validation
Algorithm
Specificationψ
“Do not let the pendulum tip over.”
“Do not collide with other aircraft.”
“Stop when the traffic light is red.”
“Do not lose more than $1 million.”
“Do not produce harmful speech.”
“Do not use | instead of \mid when writing conditional probabilities in LaTeX.”
— Mykel
Specificationψ
“Do not let the pendulum tip over.”
“Do not collide with other aircraft.”
“Stop when the traffic light is red.”
“Do not lose more than $1 million.”
“Do not produce harmful speech.”
“Do not use | instead of \mid when writing conditional probabilities in LaTeX.”
— Mykel
Formal Specification Language(e.g. Linear Temporal Logic or Signal Temporal Logic)
Formal Specification
Specificationψ
“Do not let the pendulum tip over.”
Formal Specification Language(e.g. Linear Temporal Logic or Signal Temporal Logic)
Formal Specification
“The absolute value of θ must always be less than π/4.”
Stay tuned for Lecture 4 for more details!
System
Specification
Validation
Algorithm
Validation
Algorithm
Failure Analysis
Formal Methods
Other Useful Stuff
Validation
Algorithm
Failure Analysis
Formal Methods
Other Useful Stuff
Falsification
Chapters 4 and 5
Project 1
Failure Distribution
Chapter 6
Failure Probability Estimation
Chapter 7
Project 2
Reachability for Linear Systems
Chapter 8
Project 3
Reachability for Nonlinear Systems
Chapter 9
Project 3
Reachability for Discrete Systems
Chapter 10
Project 3
Explainability
Chapter 11
Runtime Monitoring
Chapter 12
Validation
Algorithm
Failure Analysis
Formal Methods
Other Useful Stuff
Falsification
Chapters 4 and 5
Project 1
Failure Distribution
Chapter 6
Failure Probability Estimation
Chapter 7
Project 2
Reachability for Linear Systems
Chapter 8
Project 3
Reachability for Nonlinear Systems
Chapter 9
Project 3
Reachability for Discrete Systems
Chapter 10
Project 3
Explainability
Chapter 11
Runtime Monitoring
Chapter 12
Failure Analysis
Perception noiseσ = 0.25
Number of rolloutsm = 50
Pfail ≈ 0 failures50 trajectories =0.000
Validation
Algorithm
Failure Analysis
Formal Methods
Other Useful Stuff
Falsification
Chapters 4 and 5
Project 1
Failure Distribution
Chapter 6
Failure Probability Estimation
Chapter 7
Project 2
Reachability for Linear Systems
Chapter 8
Project 3
Reachability for Nonlinear Systems
Chapter 9
Project 3
Reachability for Discrete Systems
Chapter 10
Project 3
Explainability
Chapter 11
Runtime Monitoring
Chapter 12
Validation
Algorithm
Formal Methods
some math
+2 right
+1 up
Chapter 8
Validation
Algorithm
Formal Methods
bounded perception noise
some fancy math
Validation
Algorithm
Formal Methods
bounded perception noise
some fancy math
some fancy math
again
some fancy math
again
Validation
Algorithm
Formal Methods
x’ = 0.05max(0.2max(-0.24x, 0) - 0.34max(1.2x - 1.41, 0) - 0.46max(0.01x - 0.31, 0) - 0.21max(1.11x - 3.16, 0) - 0.42max(0.45x - 5.99, 0), 0) + 1.64max(0.14max(-0.24x, 0) + 0.33max(1.2x - 1.41, 0) + 0.55max(0.01x - 0.31, 0) - 0.2max(1.11x - 3.16, 0) - 2.66max(0.45x - 5.99, 0) + 0.85, 0) - 4.85max(-0.08max(-0.24x, 0) + 0.42max(1.2x - 1.41, 0) - 0.68max(0.01x - 0.31, 0) - 0.5max(1.11x - 3.16, 0) - 1.26max(0.45x - 5.99, 0) + 3.76, 0) - 0.63max(0.33max(-0.24x, 0) + 0.17max(1.2x - 1.41, 0) + 0.22max(0.01x - 0.31, 0) - 0.33max(1.11x - 3.16, 0) - 0.13max(0.45x - 5.99, 0) - 0.12, 0) + 1.77max(-0.41max(-0.24x, 0) + 1.44max(1.2x - 1.41, 0) - 0.18max(0.01x - 0.31, 0) + 1.16max(1.11x - 3.16, 0) + 1.09max(0.45x - 5.99, 0) - 1.11, 0) - 0.46
some fancy math
+ some extra tricks
Chapter 9
Validation
Algorithm
Failure Analysis
Formal Methods
Other Useful Stuff
Falsification
Chapters 4 and 5
Project 1
Failure Distribution
Chapter 6
Failure Probability Estimation
Chapter 7
Project 2
Reachability for Linear Systems
Chapter 8
Project 3
Reachability for Nonlinear Systems
Chapter 9
Project 3
Reachability for Discrete Systems
Chapter 10
Project 3
Explainability
Chapter 11
Runtime Monitoring
Chapter 12
Validation
Algorithm
Other Useful Stuff
Explainability
Chapter 11
Runtime Monitoring
Chapter 12
MIT Technology ReviewJan 2026
Mechanistic interpretability: 10 Breakthrough Technologies 2026
Sparse autoencoders and circuit tracing are starting to reveal the concepts and reasoning steps inside AI models.
Waymo BlogFeb 17, 2026
Advice, not control: the role of Remote Assistance in Waymo’s operations
When the car recognizes a situation it isn’t sure how to handle, it pings a remote human for advice — about 70 agents on call for a 3,000-car fleet.
Validation
Algorithm
Failure Analysis
Formal Methods
Other Useful Stuff
Falsification
Chapters 4 and 5
Project 1
Failure Distribution
Chapter 6
Failure Probability Estimation
Chapter 7
Project 2
Reachability for Linear Systems
Chapter 8
Project 3
Reachability for Nonlinear Systems
Chapter 9
Project 3
Reachability for Discrete Systems
Chapter 10
Project 3
Explainability
Chapter 11
Runtime Monitoring
Chapter 12
System
Specification
Validation
Algorithm
Plan for Today
1
Course Staff Intros
2
Convince You That Safety Validation Is Important
3
Introduce Validation Framework for the Course
4
Course Logistics
Course Logistics
Course Logistics
Resources
Schedule
Lectures
Textbook
Grading
Quizzes
Projects
Late Policy
Final Project
Ed Discussion
Office Hours
Miscellaneous
Course Logistics
Resources
Schedule
Lectures
Textbook
Grading
Quizzes
Projects
Late Policy
Final Project
Ed Discussion
Office Hours
Miscellaneous
You can find all of the information I am about to tell you on the course website!
aa228v.stanford.edu
QR code: aa228v.stanford.edu
Course discussions will be held on Ed.
Submit assignments on Gradescope.
Access lecture recordings on Canvas.
* Let us know on Ed if you are having trouble accessing any of these resources (or email smkatz@stanford.edu if Ed is the one you are having trouble accessing 🙂)
Course Logistics
Resources
Schedule
Lectures
Textbook
Grading
Quizzes
Projects
Late Policy
Final Project
Ed Discussion
Office Hours
Miscellaneous
Access here (also linked on the course webpage)
Course Logistics
Resources
Schedule
Lectures
Textbook
Grading
Quizzes
Projects
Late Policy
Final Project
Ed Discussion
Office Hours
Miscellaneous
T/Th 4:30–5:50pm in CoDa B80
Recordings will be posted on Canvas — attendance is not mandatory.
That said…
Please come if you can!
Note: the main source of lecture notes is the textbook.
Slides are online here, linked on the course website.
Course Logistics
Resources
Schedule
Lectures
Textbook
Grading
Quizzes
Projects
Late Policy
Final Project
Ed Discussion
Office Hours
Miscellaneous
Algorithms for Validation
algorithmsbook.com/validation
QR code: algorithmsbook.com/validation
Let us know of any bugs you find — we will add you to the acknowledgements!
Course Logistics
Resources
Schedule
Lectures
Textbook
Grading
Quizzes
Projects
Late Policy
Final Project
Ed Discussion
Office Hours
Miscellaneous
Projects: 34%
•1% Project 0
•11% Project 1
•11% Project 2
•11% Project 3
Quizzes: 31%
•1% Quiz 0
•15% Quiz 1
•15% Quiz 2
Final Project: 35%
•5% Proposal
•5% Status Update
•5% Lightning Talk
•15% Final Report
•5% Peer Review
*If you are taking the course for 4 units instead of 3, you are expected to spend an extra 30 hours on your final project.
Course Logistics
Resources
Schedule
Lectures
Textbook
Grading
Quizzes
Projects
Late Policy
Final Project
Ed Discussion
Office Hours
Miscellaneous
Quizzes are in person, during class.
Allowed
Your cheat sheet.
No AI, internet, or collaborators.
Conflict?
Take it another evening.
Coordinate with course staff.
CGOE?
Coordinate with CGOE to set up an exam monitor.
Details on the course website.
Course Logistics
Resources
Schedule
Lectures
Textbook
Grading
Quizzes
Projects
Late Policy
Final Project
Ed Discussion
Office Hours
Miscellaneous
Project 0: getting set up with the AA228V project environment
Project 1: efficient falsification
Project 2: efficient failure probability estimation
Project 3: reachability analysis
All projects have an autograded code portion and a write-up portion (except Project 0).
Submit on Gradescope.
Course Logistics
Resources
Schedule
Lectures
Textbook
Grading
Quizzes
Projects
Late Policy
Final Project
Ed Discussion
Office Hours
Miscellaneous
The projects will use the Julia programming language in Pluto notebooks.
For those not familiar with Julia, no worries! Robert has put a lot of effort into making the transition as seamless as possible.
Projects 1, 2, and 3 have a leaderboard. Winners will get a prize!
Your position on the leaderboard will not affect your grade.
Course Logistics
Resources
Schedule
Lectures
Textbook
Grading
Quizzes
Projects
Late Policy
Final Project
Ed Discussion
Office Hours
Miscellaneous
I thought long and hard about coding agents.
My decision: embrace them.
You still need to pass the autograder,
but it will be down-weighted.
Instead of a write-up, you will record a 2-minute video
explaining your solution to each part of the project.
If you can explain it, you can use it.
Course Logistics
Resources
Schedule
Lectures
Textbook
Grading
Quizzes
Projects
Late Policy
Final Project
Ed Discussion
Office Hours
Miscellaneous
All assignments are due at 5pm on Fridays.
There is a 72-hour no-questions-asked penalty-free extension applied automatically!
*Late submissions on projects will not qualify for the leaderboard.
Course Logistics
Resources
Schedule
Lectures
Textbook
Grading
Quizzes
Projects
Late Policy
Final Project
Ed Discussion
Office Hours
Miscellaneous
Final Project: 35%
•5% Proposal
•5% Status Update
•5% Lightning Talk
•15% Final Report
•5% Peer Review
Open-ended project to explore topics in greater depth.
Having trouble coming up with a topic?
•If you took AA228, maybe you can validate the decision-making system you created for that final project!
•Come to office hours to brainstorm ideas.
•Look at projects from last year.
Course Logistics
Resources
Schedule
Lectures
Textbook
Grading
Quizzes
Projects
Late Policy
Final Project
Ed Discussion
Office Hours
Miscellaneous
Ed is the fastest way to reach the course staff.
Make posts public if the answer would benefit the rest of the class.
Can make private posts as well.
Course Logistics
Resources
Schedule
Lectures
Textbook
Grading
Quizzes
Projects
Late Policy
Final Project
Ed Discussion
Office Hours
Miscellaneous
Office hours are on the course calendar (some virtual, some in person).
Come see us!
Feel free to contact me if you want to set up an individual meeting at a different time.
Course Logistics
Resources
Schedule
Lectures
Textbook
Grading
Quizzes
Projects
Late Policy
Final Project
Ed Discussion
Office Hours
Miscellaneous
Give us feedback at any time!
The course staff has done as much offline validation as we can to ensure that you have a great experience.
… but we also know that runtime monitoring is important.
Plan for Today
1
Course Staff Intros
2
Convince You That Safety Validation Is Important
3
Introduce Validation Framework for the Course
4
Course Logistics
TLDR
Validation is interesting and important — this class will give you valuable skills!
Get started on Quiz 0 and Project 0, due a week from Friday at 5pm.
Yay!