2

We have two ER-X connected via an unmanaged switch to a Cisco DPC3010 modem. Each is assigned a static WAN IP in a different block: 174.79.186.x/27 and 98.187.238.x/27. Every once in a while (roughly once a month), we will lose connection on the second (98.187.238.x/27) IP, requiring a call to our ISP (Cox) to clear the ARP cache, at which point the connection resumes. Could this be a configuration problem on our end, or is the issue solely with the ISP?

EDIT: running a tcpdump on both routers, I can see ARP requests for the 98.187.238.x/27 address, and the appropriate router responds, but apparently Cox is not seeing the response, as the request appears every few seconds:

Request who-has 98.187.238.xx tell 98.187.238.65, length 46
78:8a:20:0d:5d:92 > 1c:6a:7a:80:d2:d9, ARP, length 42: Reply 98.187.238.xx is-at 78:8a:20:0d:5d:92, length 28

Network diagram

Router configs:

firewall {
    all-ping enable
    broadcast-ping disable
    group {
    }
    ipv6-receive-redirects disable
    ipv6-src-route disable
    ip-src-route disable
    log-martians enable
    name WAN_IN {
        default-action drop
        description "WAN to internal"
        rule 10 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
    }
    name WAN_LOCAL {
        default-action drop
        description "WAN to router"
        rule 10 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
        rule 21 {
            action accept
            description "Allow eSP"
            log disable
            protocol esp
        }
        rule 22 {
            action accept
            description "Allow L2TP"
            destination {
                port 500,1701,4500
            }
            log disable
            protocol udp
        }
        rule 23 {
            action accept
            description "Allow GRE"
            destination {
            }
            log disable
            protocol gre
        }
        rule 24 {
            action accept
            description "Allow openVPN"
            destination {
                group {
                }
                port 1194
            }
            log disable
            protocol tcp_udp
            state {
                established enable
                invalid disable
                new enable
                related enable
            }
        }
        rule 25 {
            action accept
            destination {
                group {
                    address-group ADDRv4_eth0
                }
            }
            log disable
            protocol icmp
        }
    }
    receive-redirects disable
    send-redirects enable
    source-validation disable
    syn-cookies enable
}
interfaces {
    bridge br0 {
        aging 300
        bridged-conntrack disable
        hello-time 2
        max-age 20
        priority 32768
        promiscuous disable
        stp false
    }
    ethernet eth0 {
        address 174.79.186.xx/27
        description Internet
        duplex auto
        firewall {
            in {
                name WAN_IN
            }
            local {
                name WAN_LOCAL
            }
        }
        mac C0:4A:00:66:49:51
        speed auto
    }
    ethernet eth1 {
        description Local
        duplex auto
        speed auto
    }
    ethernet eth2 {
        description Local
        duplex auto
        speed auto
    }
    ethernet eth3 {
        bridge-group {
            bridge br0
        }
        description Local
        duplex auto
        speed auto
    }
    ethernet eth4 {
        description Local
        duplex auto
        speed auto
    }
    loopback lo {
    }
    openvpn vtun0 {
        bridge-group {
            bridge br0
            cost 100
        }
        description "Pascal phone"
        device-type tap
        encryption aes256
        local-host 174.79.186.xx
        local-port 1194
        mode site-to-site
        openvpn-option --float
        openvpn-option "--auth none"
        openvpn-option "--ping 5"
        remote-port 1194
        shared-secret-key-file *REMOVED*
    }
    switch switch0 {
        address 192.168.1.1/24
        description Local
        mtu 1500
        switch-port {
            interface eth1 {
            }
            interface eth2 {
            }
            interface eth4 {
            }
            vlan-aware disable
        }
    }
}
port-forward {
    auto-firewall enable
    hairpin-nat enable
    lan-interface switch0
    lan-interface eth4.2
    wan-interface eth0
}
protocols {
    static {
    }
}
service {
    dhcp-server {
        disabled false
        hostfile-update disable
        shared-network-name LAN {
            authoritative disable
            subnet 192.168.1.0/24 {
                default-router 192.168.1.1
                dns-server 8.8.8.8
                dns-server 8.8.4.4
                lease 7200
                start 192.168.1.51 {
                    stop 192.168.1.100
                }
                static-mapping Canon_Pixma_MX452 {
                    ip-address 192.168.1.75
                    mac-address 18:0c:ac:f8:0d:4c
                }
                static-mapping SCOTT-PC {
                    ip-address 192.168.1.59
                    mac-address 00:50:8d:9d:3b:f0
                }
            }
        }
        use-dnsmasq disable
    }
    dns {
        forwarding {
            cache-size 150
            listen-on switch0
            listen-on eth4.2
        }
    }
    gui {
        http-port 80
        https-port 443
        older-ciphers enable
    }
    nat {
        rule 5010 {
            description "masquerade for WAN"
            log disable
            outbound-interface eth0
            protocol all
            source {
                address 192.168.1.0/24
            }
            type masquerade
        }
    }
    ssh {
        port 22
        protocol-version v2
    }
    upnp {
        listen-on switch0 {
            outbound-interface eth0
        }
    }
}
system {
    gateway-address 174.79.186.33
    host-name ER-XP
    login {
        user admin {
            authentication {
                encrypted-password ****************
                plaintext-password ****************
            }
            full-name *REMOVED*
            level admin
        }
    }
    name-server 8.8.8.8
    name-server 8.8.4.4
    ntp {
        server 0.ubnt.pool.ntp.org {
        }
        server 1.ubnt.pool.ntp.org {
        }
        server 2.ubnt.pool.ntp.org {
        }
        server 3.ubnt.pool.ntp.org {
        }
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone America/New_York
}
traffic-control {
}

and

firewall {
    all-ping enable
    broadcast-ping disable
    ipv6-receive-redirects disable
    ipv6-src-route disable
    ip-src-route disable
    log-martians enable
    name LAN_LOCAL {
        default-action accept
        description "Local network isolation"
        rule 1 {
            action drop
            destination {
                address 192.168.1.xx
            }
            log disable
            protocol !icmp
            source {
                address 192.168.44.0/24
            }
        }
    }
    name WAN_IN {
        default-action drop
        description "WAN to internal"
        rule 10 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
    }
    name WAN_LOCAL {
        default-action drop
        description "WAN to router"
        rule 10 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 30 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
        rule 31 {
            action accept
            description "Allow ICMP"
            destination {
                group {
                    address-group ADDRv4_eth0
                }
            }
            icmp {
                type 8
            }
            log disable
            protocol icmp
        }
    }
    receive-redirects disable
    send-redirects enable
    source-validation disable
    syn-cookies enable
}
interfaces {
    ethernet eth0 {
        address 98.187.238.xx/27
        description Internet
        duplex auto
        firewall {
            in {
                name WAN_IN
            }
            local {
                name WAN_LOCAL
            }
        }
        speed auto
    }
    ethernet eth1 {
        address 192.168.1.xx/24
        description Mgmt
        duplex auto
        speed auto
    }
    ethernet eth2 {
        description Local
        duplex auto
        speed auto
    }
    ethernet eth3 {
        description Local
        duplex auto
        speed auto
    }
    ethernet eth4 {
        description Local
        duplex auto
        speed auto
    }
    loopback lo {
    }
    switch switch0 {
        address 192.168.44.1/24
        description Webserver_network
        firewall {
            local {
                name LAN_LOCAL
            }
        }
        mtu 1500
        switch-port {
            interface eth2 {
            }
            interface eth3 {
            }
            interface eth4 {
            }
            vlan-aware disable
        }
    }
}
port-forward {
    auto-firewall enable
    hairpin-nat enable
    lan-interface switch0
    rule 1 {
        description HTTPS
        forward-to {
            address 192.168.44.xx
            port 443
        }
        original-port 443
        protocol tcp
    }
    rule 2 {
        description SSH
        forward-to {
            address 192.168.44.xx
            port 22
        }
        original-port *REMOVED*
        protocol tcp_udp
    }
    rule 3 {
        description "HTTP for LE"
        forward-to {
            address 192.168.44.xx
            port 80
        }
        original-port 80
        protocol tcp_udp
    }
    wan-interface eth0
}
service {
    dns {
        forwarding {
            cache-size 150
            listen-on eth1
            listen-on switch0
        }
    }
    gui {
        http-port 80
        https-port 443
        listen-address 192.168.1.xx
        older-ciphers enable
    }
    nat {
        rule 5010 {
            description "masquerade for WAN"
            outbound-interface eth0
            type masquerade
        }
    }
    ssh {
        listen-address 192.168.1.xx
        port 22
        protocol-version v2
    }
    ubnt-discover {
        disable
    }
    unms {
        disable
    }
}
system {
    gateway-address 98.187.238.65
    host-name USXR
    login {
        user *REMOVED* {
            authentication {
                encrypted-password ****************
            }
            level admin
        }
    }
    name-server 8.8.8.8
    name-server 8.8.4.4
    ntp {
        server 0.ubnt.pool.ntp.org {
        }
        server 1.ubnt.pool.ntp.org {
        }
        server 2.ubnt.pool.ntp.org {
        }
        server 3.ubnt.pool.ntp.org {
        }
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone America/New_York
}
5
  • 1
    "Could this be a configuration problem on our end...?" Without seeing your configuration, it's hard to tell. – Ron Trunk Feb 7 '18 at 16:20
  • What configuration information do you need? – Scott Feb 7 '18 at 17:22
  • 1
    The routers and switch configurations. – Ron Trunk Feb 7 '18 at 18:02
  • 1
    You need to post all device configurations.Where is the ARP cache cleared? The SG110D is for WAN? – Zac67 Feb 7 '18 at 18:02
  • The ARP cache is cleared somewhere on the ISP's router, since resetting the cable modem does nothing. The SG110D is an unmanaged switch. Rather than deal with DNAT, I'm trying to use two routers to isolate the separate IPs. – Scott Feb 7 '18 at 18:54
2

I now believe the 192.168.44.0 router is defective for two reasons:

  1. It takes almost 30 seconds to login to the cli on the web-gui. Other identical routers have no such delay.

  2. Our ISP was able to find an unused address adjacent to our existing main address in the 174.79.186.xx/27 block. When I configured that address on the router with the 192.168.44.0/24 subnet, it still failed to respond to ARP requests and never saw ARP responses from the gateway. Oddly enough, when configured with the old address, it worked briefly.

EDIT: (2/10/2018)

Today I factory reset the 192.168.44.0 without restoring the config and ran the basic wizard on it, supplying the newer static IP. It refuses to see the gateway's ARP responses:

Request who-has 174.79.186.33 tell 174.79.186.xx, length 46

Supposedly it responds to the gateway's requests:

Request who-has 174.79.186.xx tell 174.79.186.33, length 46
78:8a:20:0d:5d:92 > 1c:6a:7a:80:d2:d9, ARP, length 42: Reply 174.79.186.xx is-at 78:8a:20:0d:5d:92, length 28

but without success, as the above entry gets repeated multiple times per minute.

1

From the info provided in the update this really seems to be an issue ISP side.

However, I would contact the ISP support with factual information, explaining what you told us here (connection drop this often, come back when they reset their equipment), provide them the tcpdump, and ask them to assist you in troubleshooting the issue. Then let them draw their own conclusion.

This approach is often more successful than saying straight away "fix you damn stuff". And some times, it saves your face when it appears that the issue was actually your side.

1
  • Oddly enough, this time, having them clear the ARP cache did not work. Just got off the phone with a tech - he stated the DPC3010 does not support two different subnets and suggested we switch to a new subnet with contiguous addresses. – Scott Feb 8 '18 at 16:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.