Understanding their environment is one part of the equation. But self-driving cars also need to know their current state of motion.
After all, seeing a stop sign doesn't help unless you know how fast you are traveling and how to break without spilling coffee everywhere.