I've been trying to figure this out but all the documentation I can find seems to be very old. The feeling seems to be that once Dell acquired Sonicwall, documentation became sparse and mostly useless. Ended up here at StackExchange due to a few Google searches bringing me to conversations where it looked like people were doing this sort of thing recently and knew what they were doing.

This was the main piece of documentation that I was using for reference, but as you can see it dates to 2006 and is for SonicOS Enhanced 3. SonicwallOS Enhanced to Openswan Linux Using GroupVPN with XAUTH.

Here's the environment

Sonicwall (SonicOS Enhanced - internet - Netgear router - Cisco integrated business router, series 881 (In the DMZ for the Netgear router) - Ubuntu 14.04 running Openswan.

LAN subnet on Sonicwall:
Lan behind Cisco:
Ubuntu private IP:

So where do I even start? I configured ipsec.conf file as described in that PDF:

conn TSBackup

On the Sonicwall, there's a Site-to-Site defined with the following settings:

Policy type: Site to site
Auth method: IKE using Preshared Secret
IPsec Primary gateway: (public IP of Ubuntu server)
Local IKE ID: (Sonicwall's ID)
Peer IKE ID: ubuntuid
Local network: Lan Subnets
Remote Network: (Address object, pointing to Ubuntu's private address
Phase 1: Main Mode, Group 2, 3DES-Sha1 28800
Phase 2: ESP, 3DES, SHA1, perfect forward secret not checked,28800
Enable Keep alive
Dead peer detection: 180, 3
VPN policy bound to Zone WAN

I don't even know where to look for logs. I've started over too, thinking maybe I goofed somewhere. Any help or advice is greatly appreciated, and my apologies for being green in all of this.

Results when initiating the connection:

104 "TSBackup" #1: STATE_MAIN_I1: initiate
003 "TSBackup" #1: ignoring unknown Vendor ID payload [5b362bc820f60008]
003 "TSBackup" #1: received Vendor ID payload [RFC 3947] method set to=115
106 "TSBackup" #1: STATE_MAIN_I2: sent MI2, expecting MR2
003 "TSBackup" #1: ignoring Vendor ID payload [Sonicwall 1 (TZ 170 Standard?)]
003 "TSBackup" #1: received Vendor ID payload [XAUTH]
003 "TSBackup" #1: received Vendor ID payload [Dead Peer Detection]
003 "TSBackup" #1: NAT-Traversal: Result using draft-ietf-ipsec-nat-t-ike (MacOS X): i am NATed
108 "TSBackup" #1: STATE_MAIN_I3: sent MI3, expecting MR3
003 "TSBackup" #1: next payload type of ISAKMP Hash Payload has an unknown value: 175
003 "TSBackup" #1: malformed payload in packet
010 "TSBackup" #1: STATE_MAIN_I3: retransmission; will wait 20s for response
003 "TSBackup" #1: ignoring informational payload, type INVALID_COOKIE msgid=00000000
003 "TSBackup" #1: received and ignored informational message

ipsec barf @ Pastebin

Edit: Updated ipsec.conf and SonicWall settings. Added results from opening connection.

Edit 2: addded link to ipsec barf.

  • Are you trying to connect just to the ubuntu server or bridge networks? Your Remote Network Address Object on the Sonicwall should be a 'network' address object for the network if you wish to bridge networks. Also when referring to the subnets in the ipsec.conf file I'd use the .0/24 address there as well. – Mike Naylor Aug 18 '14 at 15:58
  • I'm only intending for the ubuntu server to connect. As for the subnet on the Sonicwall's side, the Lan subnet is Is there some weirdness considering I'm connecting to a supernet? – toxeia Aug 18 '14 at 16:41
  • I've made some progress, but I'm currently wrestling with a message stating OpenSwan has received a malformed payload. There also seems to be a mention of information messages that are being ignored - and frustratingly not logged. – toxeia Aug 22 '14 at 18:58
  • It sounds like you are trying to connect to the site-to-site VPN, but WAN Group VPN is not a site-to-site VPN configuration. The Group VPN is designed for use with the Global VPN Client, so you may have better luck following instructions for getting the Global VPN client working with your SonicWALL. Alternatively, the SSLVPN is Java based and works on linux, so you could try that instead. – James Shewey Mar 23 '16 at 2:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.