How to solve a BKN* Port Issue on a cisco catalyst

Question

There is a Cisco ISR 4451 connected with Portchannel to a Catalyst 3850 Stack and there are some VLANs enabled, but only one does not come up for a Portinconsistence. I figure out an Issue on the equipment behind my catalyst stack but do not know how to debug this issue.

For better understanding here is a simple sketch of the Network:

There are three VLANs on the ISR and Catalyst. The Portchannel act as a trunk and the Interfaces use access VLAN and one VLAN is for native VLAN. The VLAN 410 works fine but one VLAN 409 I get a type inconsistent error.

*%SPANTREE-7-RECV_1Q_NON_TRUNK: Received 802.1Q BPDU on non-trunk GigabitEthernet1/0/3 VLAN409.
*%SPANTREE-7-BLOCK_PORT_TYPE: Blocking GigabitEthernet1/0/3 on VLAN0409. Inconsistent port type.

Here is the config from catalyst Ports:

interface Port-channel2
 switchport trunk native vlan 10
 switchport mode trunk
end
interface GigabitEthernet1/0/2
 description PortChannel ISR
 switchport trunk native vlan 10
 switchport mode trunk
 channel-group 2 mode on
end
interface GigabitEthernet1/0/3
 switchport access vlan 409
end
interface GigabitEthernet1/0/4
 switchport access vlan 410
end

And this is the config of the ISR Ports

interface GigabitEthernet0/0/1
 no ip address
 media-type sfp
 negotiation auto
 channel-group 2
end
interface Port-channel2.409
 encapsulation dot1Q 409
 ip address 10.1.18.5 255.255.255.252
end
interface Port-channel2.410
 encapsulation dot1Q 410
 ip address 10.1.18.1 255.255.255.252
end

Both interfaces are configured the same way, one works and the other one not. How can I debug this case and solve this issue?

The Output of sh spanning-tree show me the Port is broken, but I do not know why. Maybe any misconfiguration on the other side of the link? sh spanning-tree vlan 409

VLAN0409
  Spanning tree enabled protocol rstp
  Root ID    Priority    33176
             Address     00a2.89b2.0f80
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    33176  (priority 32768 sys-id-ext 409)
             Address     00a2.89b2.0f80
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/3             Desg BKN*4         128.3    P2p *TYPE_Inc 
Po2                 Desg FWD 3         128.2316 P2p 

UPDATE

complete Configuration:

Building configuration...

Current configuration : 9280 bytes
!
! Last configuration change at 12:08:55 UTC Tue Sep 11 2018
!
version 15.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
service compress-config
!
hostname cat03
!
boot-start-marker
boot-end-marker
!
!
vrf definition Mgmt-vrf
 !
 address-family ipv4
 exit-address-family
 !
 address-family ipv6
 exit-address-family
!
!
no aaa new-model
switch 1 provision ws-c3850-12s
switch 2 provision ws-c3850-12s
!
!
!
!
!
!
!
!
!
!
ip domain-name demo.de
!
!
qos queue-softmax-multiplier 100
!
!
!
errdisable recovery cause udld
errdisable recovery cause bpduguard
errdisable recovery cause security-violation
errdisable recovery cause channel-misconfig
errdisable recovery cause pagp-flap
errdisable recovery cause dtp-flap
errdisable recovery cause link-flap
errdisable recovery cause sfp-config-mismatch
errdisable recovery cause gbic-invalid
errdisable recovery cause l2ptguard
errdisable recovery cause psecure-violation
errdisable recovery cause port-mode-failure
errdisable recovery cause dhcp-rate-limit
errdisable recovery cause pppoe-ia-rate-limit
errdisable recovery cause mac-limit
errdisable recovery cause vmps
errdisable recovery cause storm-control
errdisable recovery cause inline-power
errdisable recovery cause arp-inspection
errdisable recovery cause loopback
errdisable recovery cause psp
errdisable recovery interval 60
diagnostic bootup level minimal
spanning-tree mode rapid-pvst
spanning-tree extend system-id
hw-switch switch 1 logging onboard message level 3
hw-switch switch 2 logging onboard message level 3
!
redundancy
 mode sso
!
!
vlan configuration 100,408-410
!
!
class-map match-any non-client-nrt-class
!
policy-map port_child_policy
 class non-client-nrt-class
  bandwidth remaining ratio 10
!
! 
!
!
!
!
!
!
!
!
!
!
!
!
interface Port-channel2
 switchport trunk native vlan 10
 switchport mode trunk
!
interface GigabitEthernet0/0
 vrf forwarding Mgmt-vrf
 ip address 10.1.20.60 255.255.255.0
 negotiation auto
!
interface GigabitEthernet1/0/1
!
interface GigabitEthernet1/0/2
 description PortChannel ISR
 switchport trunk native vlan 10
 switchport mode trunk
 channel-group 2 mode on
!
interface GigabitEthernet1/0/3
 switchport access vlan 409
 spanning-tree bpduguard disable
!
interface GigabitEthernet1/0/4
 switchport access vlan 410
!
interface GigabitEthernet1/0/5
!
interface GigabitEthernet1/0/6
!
interface GigabitEthernet1/0/7
!
interface GigabitEthernet1/0/8
!
interface GigabitEthernet1/0/9
!
interface GigabitEthernet1/0/10
!
interface GigabitEthernet1/0/11
!
interface GigabitEthernet1/0/12
!
interface GigabitEthernet1/1/1
!
interface GigabitEthernet1/1/2
!
interface GigabitEthernet1/1/3
!
interface GigabitEthernet1/1/4
!
interface TenGigabitEthernet1/1/1
!
interface TenGigabitEthernet1/1/2
!
interface TenGigabitEthernet1/1/3
!
interface TenGigabitEthernet1/1/4
!
interface GigabitEthernet2/0/1
!
interface GigabitEthernet2/0/2
!
interface GigabitEthernet2/0/3
!         
interface GigabitEthernet2/0/4
!
interface GigabitEthernet2/0/5
!
interface GigabitEthernet2/0/6
!
interface GigabitEthernet2/0/7
!
interface GigabitEthernet2/0/8
!
interface GigabitEthernet2/0/9
!
interface GigabitEthernet2/0/10
!
interface GigabitEthernet2/0/11
!
interface GigabitEthernet2/0/12
!
interface GigabitEthernet2/1/1
!
interface GigabitEthernet2/1/2
!
interface GigabitEthernet2/1/3
!
interface GigabitEthernet2/1/4
!
interface TenGigabitEthernet2/1/1
!
interface TenGigabitEthernet2/1/2
!
interface TenGigabitEthernet2/1/3
!
interface TenGigabitEthernet2/1/4
!         
interface Vlan409
 no ip address
!
ip forward-protocol nd
ip http server
ip http authentication local
ip http secure-server
!
!
snmp-server community private RO
snmp-server trap-source GigabitEthernet0/0
!
!
line con 0
 stopbits 1
line aux 0
 stopbits 1
line vty 0 4
 exec-timeout 0 0
 login local
 transport input ssh
line vty 5 15
 exec-timeout 0 0
 login local
 transport input ssh
!         
!
wsma agent exec
 profile httplistener
 profile httpslistener
!
wsma agent config
 profile httplistener
 profile httpslistener
!
wsma agent filesys
 profile httplistener
 profile httpslistener
!
wsma agent notify
 profile httplistener
 profile httpslistener
!
!
wsma profile listener httplistener
 transport http
!
wsma profile listener httpslistener
 transport https
!
ap group default-group
end

Answer 1

The device connected to port 1/0/3 appears to be trunking, despite what your ISP claims. The simple way to fix it is to make your interface a trunk as well:

interface gi 1/0/3
switchport mode trunk
switchport trunk allowed vlan 409
switchport trunk native vlan 409

Packets for vl 409 will pass untagged.

EDIT:

By using

debug spanning-tree

it was determined that the ISP was using a different VLAN (929)for the native VLAN despite their claim to the contrary.

So, when troubleshooting this kind of issue, the switch debug messages can give you insight into the problem. The debug messages can provide important details on exactly what the mismatch is, allowing you to adjust your configuration.