I know that to secure OSPF you should 1) use OSPF authentication, 2) use passive interface command on interfaces that do not have ospf neighbors. If I only use the passive interface command and not ospf authentication, what vulnerabilities am I left open to?
One issue is that authentication ensures that only trusted devices are capable of exchanging routes on the network. Without authentication, you could introduce a non-trusted device and cause significant routing issues. For example:
If area 0 is not authenticated, attach a router in area 0 with bogus routes to null0. You could even create a default route and inject it into the topology leading to the bad router to black hole traffic. Or the route could force traffic toward a bogus gateway designed to sniff connections and pull out insecure data before sending it on the right path.
Authentication ensures that only routers you know about and trust are exchanging information.
It depends on your network topology. If the non-passive links are isolated (point-to-point) and secured at lower layers of the stack (physical access control of the routers), then I would be hard pressed to identify a viable attack vector. Authentication is critical when it is possible for a rogue router to present arbitrary traffic on a given link.
If anyone were to gain access to the actual equipment and somehow insert another device to the far end of the link, that would give them access to your network, to inject routes into your routing table and other nasty stuff like that.
A scenario like this would be very theoretical in places such as backbone networks that are in secured locations, but should the link go to a customer or to another third party, some sort of authentication would probably be very wise.
If we make an assumption that your layers 1-3 are secure than OSPF authentication doesn't make any sense. But because layer 1-3 are not necessarily secure OSPF employs it's own security method - authentication.
Authentication in OSPF prevents an attacker who can sniff and inject packets to fool routers and modify OSPF topology. Results are for example: possible of MITM when attacker changes the topology in such way that certain/all traffic flows through machine controlled by him. Denial of service when the attacker discards traffic which flows through him. Another result could be meltdown of all routers when attacker announces new information very rapidly, although this could be partially solved by tuning SPF timers.
Authentication also prevents replay attacks, for example it prevents attacker from advertising expired information from past. Also it prevents mayhem by plugging a router from another network with existing OSPF configuration which could inject overlapping routes for example (thanks to this, authentication is good even if you have your layer 1-3 secured).
Is it possible to encrypt OSPF?
OSPFv2 only supports authentication. You can still see the LSAs payload even if you use authentication. The only thing authentication does is authenticate neighbors. There is no payload encryption.
OSPF (Open Shortest Path First) Version 2 defines the fields AuType and Authentication in its protocol header to provide security. In OSPF for IPv6 (OSPFv3), both of the authentication fields were removed from OSPF headers. OSPFv3 relies on the IPv6 Authentication Header (AH) and IPv6 Encapsulating Security Payload (ESP) to provide integrity, authentication, and/or confidentiality.
So, if OSPFv3 we can ecrypt entire packet by IPSec.
Once you have interfaces set to passive, you're not open to much. Authentication does add two possible vectors for trouble:
CPU utilization - this isn't necessarily a huge issue, but shouldn't be forgotten when you're doing your calculations. However, if you are running a network where convergence times take longer than you want, every little bit counts.
Troubleshooting. It's easy to miss something, and that can slow down bringing up a new connection, replacement router, etc.
If you're worried about getting OSPF sniffed and having a malicious intruder inject data, you should probably be running something stronger than authentication: start with running actual encryption rather than the weak MD5 you're going to get with OSPFv2, and BGP is better for untrusted links.