A network (L2 segment / IP subnet) is connected by a switch or VLAN. Routers route between networks.
An additional switch in that place allows you to connect additional devices to the transfer network. You've also added another "many ISP router", please check whether that's intentional.
There are some device connected to that additional sw which cannot be migrated or moved to other place
That's not a good reason to add device to a transfer network.
I want to make sure that when some layer 3 packet flowing through the additional sw won't block it or drop the traffic accidentally
That depends entirely on the IP configuration that you told us nothing about.
when 2 OSPF router can form neighbourhood when they both connect to the same router(same VLAN), so that the additional sw is ignored in this situation ?
OSPF uses multicast or explicit unicast to discover neighbors and form adjacencies. As long as you don't change the IP configuration, an additional switch doesn't make a difference.
From the diagram, it looks like you're connecting a switch to the routers' WAN uplinks. That doesn't make sense in the scenario that you describe.
What you seem to want(?) is to use additional, routed ports on the routers and route between them using a private transfer network - no switch necessary. With static routes on either side (or OSPF) you could transparently route between the PCs connected to both routers. If that is what you're trying to achieve.