There are two phases I've seen in becoming a security engineer. The first phase is moving beyond the "I am an engineer" mindset, where the goal is to build systems that fit into a specific set of design constraints. You have to realize that systems can operate outside of those design constraints. This is shockingly hard for good engineers to learn -- I've been in rooms full of them when they have their "aha" moment.
The second phase is when engineers realize that just whacking specific vulnerabilities is not going to end bugs -- that you need to take systematic actions to close entire vulnerability classes. That's where formal verification, sandboxing, MTE etc. come from. But in practice so far, this doesn't end vulnerabilities, it just leads to a bunch of new and more exciting ones.
I want to believe that with enough of a push we can get AIs to finish all of this and we'll be security-bug free. But if we can't, at least we can get to the point where new vulnerabilities are costly again.
I feel that it's not such a clear distinction between whackamole bug fixing and systematic security engineering. For instance, he uses an example of recurring security vulnerabilities in Chromium: a DCHECK violation again and again and again.
However, DCHECK is a Chromium assertion used to defensively check invariants. In other words, it's an example of the invariant-based security engineering, which he contrasts with the whackamole approach.
Just to be clear, I think he has a point and I enjoyed reading it---but the problems we're saddled with won't disappear in a flash of enlightenment.
While I had the same thoughts about the coming cyber-apocalypse, the ugly truth is that it doesn't make a difference as in a year or so models will be so persuasive and skilled in social engineering that even the hardest cyberdefense in the world doesn't keep them from exfiltrating any information they want by targeting not the system, but the users: leveraging data from the dark web, humans for rent or devising highly deceptive scams you can t even imagine.
Maybe you can upgrade digital systems until they are 'fully fortified' in theory, but humans have an upper limit what they can understand / perceive / critically analyse (not to even speak about engineered high pressure situations) that can't be upgraded.
> Maybe you can upgrade digital systems until they are 'fully fortified' in theory, but humans have an upper limit what they can understand / perceive / critically analyse (not to even speak about engineered high pressure situations) that can't be upgraded.
There is an answer to that - simplify and abstract. Lots of human software is unnecessarily complex, often caused by backwards compatibility and general human creativity.
Take sendmail vs postfix as an example of this process.
> leverage AI to implement verifiable enforcement of the invariants
The only verifiable invariants that are going to work in the long run are:
air gaps
data diodes
Interposing relays used to be used with PLCs[1] to prevent motor control systems from energizing both forward and reverse power to a large motor in the unlikely but possible event of control systems failure. We need to apply the same amount of engineering rigor to our computer systems.
We simply shouldn't trust software to a job that belongs in hardware.
The second phase is when engineers realize that just whacking specific vulnerabilities is not going to end bugs -- that you need to take systematic actions to close entire vulnerability classes. That's where formal verification, sandboxing, MTE etc. come from. But in practice so far, this doesn't end vulnerabilities, it just leads to a bunch of new and more exciting ones.
I want to believe that with enough of a push we can get AIs to finish all of this and we'll be security-bug free. But if we can't, at least we can get to the point where new vulnerabilities are costly again.
Maybe you can upgrade digital systems until they are 'fully fortified' in theory, but humans have an upper limit what they can understand / perceive / critically analyse (not to even speak about engineered high pressure situations) that can't be upgraded.
There is an answer to that - simplify and abstract. Lots of human software is unnecessarily complex, often caused by backwards compatibility and general human creativity.
Take sendmail vs postfix as an example of this process.
The only verifiable invariants that are going to work in the long run are:
Interposing relays used to be used with PLCs[1] to prevent motor control systems from energizing both forward and reverse power to a large motor in the unlikely but possible event of control systems failure. We need to apply the same amount of engineering rigor to our computer systems.We simply shouldn't trust software to a job that belongs in hardware.
[1] https://en.wikipedia.org/wiki/Programmable_logic_controller