[strongSwan-dev] Multiple MySQL virtual IP pools result in charon suicide

[lauri]

Hello,

I've been using virtual IP pool stored in MySQL server for a while
with StrongSwan gateway on Ubuntu 16.04 machine
(U5.3.5/K4.4.0-79-generic).

Everything worked fine until I added another pool using ipsec leases
command and reconfigured charon somewhat like this, in this case
%linux and %windows are the pools stored in MySQL:

conn linux
auto=add
right=%any
rightsourceip=%linux
left=vpn.example.com
leftcert=/etc/ipsec.d/certs/vpn.pem
leftsubnet=10.20.30.0/24
rightca="CN=ca-for-linux-boxes"

conn windows
auto=add
right=%any
rightsourceip=%windows
left=vpn.example.com
leftcert=/etc/ipsec.d/certs/vpn.pem
leftsubnet=10.20.30.0/24
rightca="CN=ca-for-windows-boxes"

It seems this is causing some sort of multithreading race condition
bug to arise which kills charon and restarts the daemon after every
couple of minutes:

vpn charon[1986]: 11[KNL] policy already exists, try to update it
vpn charon[1986]: 11[KNL] policy already exists, try to update it
vpn charon[1986]: 12[LIB] preparing MySQL statement failed: Lost
connection to MySQL server during query
vpn charon[1986]: 05[DMN] thread 5 received 11
vpn charon[1986]: 05[LIB] dumping 16 stack frame addresses:
vpn charon[1986]: 05[LIB] /lib/x86_64-linux-gnu/libpthread.so.0 @
0x7f14f34d9000 [0x7f14f34ea390]
vpn charon[1986]: 05[LIB] -> ??:?
vpn charon[1986]: 05[LIB]
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 @ 0x7f14e3388000
[0x7f14e33bbbb6]
vpn charon[1986]: 05[LIB] -> ??:?
vpn charon[1986]: 05[LIB]
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 @ 0x7f14e3388000
(mysql_ping+0x26) [0x7f14e33aeb26]
vpn charon[1986]: 05[LIB] -> ??:?
vpn charon[1986]: 05[LIB]
/usr/lib/ipsec/plugins/libstrongswan-mysql.so @ 0x7f14e3998000
[0x7f14e3999f0d]
vpn charon[1986]: 05[LIB] ->
/build/strongswan-UD5DOo/strongswan-5.3.5/src/libstrongswan/plugins/mysql/mysql_database.c:236
vpn charon[1986]: 05[LIB]
/usr/lib/ipsec/plugins/libstrongswan-mysql.so @ 0x7f14e3998000
[0x7f14e399a2de]
vpn charon[1986]: 05[LIB] ->
/build/strongswan-UD5DOo/strongswan-5.3.5/src/libstrongswan/plugins/mysql/mysql_database.c:542
vpn charon[1986]: 05[LIB]
/usr/lib/ipsec/plugins/libstrongswan-attr-sql.so @ 0x7f14e2b6b000
[0x7f14e2b6bd14]
vpn charon[1986]: 05[LIB] ->
/build/strongswan-UD5DOo/strongswan-5.3.5/src/libcharon/plugins/attr_sql/attr_sql_provider.c:93
vpn charon[1986]: 05[LIB]
/usr/lib/ipsec/plugins/libstrongswan-attr-sql.so @ 0x7f14e2b6b000
[0x7f14e2b6bec1]
vpn charon[1986]: 05[LIB] ->
/build/strongswan-UD5DOo/strongswan-5.3.5/src/libcharon/plugins/attr_sql/attr_sql_provider.c:398
vpn charon[1986]: 05[LIB] /usr/lib/ipsec/libstrongswan.so.0 @
0x7f14f3b7f000 [0x7f14f3b93e74]
vpn charon[1986]: 05[LIB] ->
/build/strongswan-UD5DOo/strongswan-5.3.5/src/libstrongswan/collections/enumerator.c:438
vpn charon[1986]: 05[LIB] /usr/lib/ipsec/libcharon.so.0 @
0x7f14f36f6000 [0x7f14f373b35d]
vpn charon[1986]: 05[LIB] ->
/build/strongswan-UD5DOo/strongswan-5.3.5/src/libcharon/sa/ikev2/tasks/ike_config.c:400
vpn charon[1986]: 05[LIB] /usr/lib/ipsec/libcharon.so.0 @
0x7f14f36f6000 [0x7f14f372fb7f]
vpn charon[1986]: 05[LIB] ->
/build/strongswan-UD5DOo/strongswan-5.3.5/src/libcharon/sa/ikev2/task_manager_v2.c:781
vpn charon[1986]: 05[LIB] /usr/lib/ipsec/libcharon.so.0 @
0x7f14f36f6000 [0x7f14f3723ff7]
vpn charon[1986]: 05[LIB] ->
/build/strongswan-UD5DOo/strongswan-5.3.5/src/libcharon/sa/ike_sa.c:1402
vpn charon[1986]: 05[LIB] /usr/lib/ipsec/libcharon.so.0 @
0x7f14f36f6000 [0x7f14f371c981]
vpn charon[1986]: 05[LIB] ->
/build/strongswan-UD5DOo/strongswan-5.3.5/src/libcharon/processing/jobs/process_message_job.c:74
vpn charon[1986]: 05[LIB] /usr/lib/ipsec/libstrongswan.so.0 @
0x7f14f3b7f000 [0x7f14f3bacb3b]
vpn charon[1986]: 05[LIB] ->
/build/strongswan-UD5DOo/strongswan-5.3.5/src/libstrongswan/processing/processor.c:235
vpn charon[1986]: 05[LIB] /usr/lib/ipsec/libstrongswan.so.0 @
0x7f14f3b7f000 [0x7f14f3bbd89c]
vpn charon[1986]: 05[LIB] ->
/build/strongswan-UD5DOo/strongswan-5.3.5/src/libstrongswan/threading/thread.c:304
(discriminator 3)
vpn charon[1986]: 05[LIB] /lib/x86_64-linux-gnu/libpthread.so.0 @
0x7f14f34d9000 [0x7f14f34e06ba]
vpn charon[1986]: 05[LIB] -> ??:?
vpn charon[1986]: 05[LIB] /lib/x86_64-linux-gnu/libc.so.6 @
0x7f14f3110000 (clone+0x6d) [0x7f14f321682d]
vpn charon[1986]: 05[LIB] -> ??:?
vpn charon[1986]: 05[DMN] killing ourself, received critical signal
vpn ipsec_starter[32468]: charon has died -- restart scheduled (5sec)

Note that MySQL server is connected over the network, it's not on the
local machine if that's relevant.

[lauri]

Hi again,

I can confirm that same setup with local MySQL server doesn't trigger
the bug. I basically moved the database to local MySQL instance and
the problems are gone. I suspect StrongSwan unittests don't consider
remote MySQL servers?

Post by lauri
Hello,
I've been using virtual IP pool stored in MySQL server for a while
with StrongSwan gateway on Ubuntu 16.04 machine
(U5.3.5/K4.4.0-79-generic).
Everything worked fine until I added another pool using ipsec leases
command and reconfigured charon somewhat like this, in this case
conn linux
auto=add
right=%any
rightsourceip=%linux
left=vpn.example.com
leftcert=/etc/ipsec.d/certs/vpn.pem
leftsubnet=10.20.30.0/24
rightca="CN=ca-for-linux-boxes"
conn windows
auto=add
right=%any
rightsourceip=%windows
left=vpn.example.com
leftcert=/etc/ipsec.d/certs/vpn.pem
leftsubnet=10.20.30.0/24
rightca="CN=ca-for-windows-boxes"
It seems this is causing some sort of multithreading race condition
bug to arise which kills charon and restarts the daemon after every
vpn charon[1986]: 11[KNL] policy already exists, try to update it
vpn charon[1986]: 11[KNL] policy already exists, try to update it
vpn charon[1986]: 12[LIB] preparing MySQL statement failed: Lost
connection to MySQL server during query
vpn charon[1986]: 05[DMN] thread 5 received 11
0x7f14f34d9000 [0x7f14f34ea390]
vpn charon[1986]: 05[LIB] -> ??:?
vpn charon[1986]: 05[LIB]
[0x7f14e33bbbb6]
vpn charon[1986]: 05[LIB] -> ??:?
vpn charon[1986]: 05[LIB]
(mysql_ping+0x26) [0x7f14e33aeb26]
vpn charon[1986]: 05[LIB] -> ??:?
vpn charon[1986]: 05[LIB]
[0x7f14e3999f0d]
vpn charon[1986]: 05[LIB] ->
/build/strongswan-UD5DOo/strongswan-5.3.5/src/libstrongswan/plugins/mysql/mysql_database.c:236
vpn charon[1986]: 05[LIB]
[0x7f14e399a2de]
vpn charon[1986]: 05[LIB] ->
/build/strongswan-UD5DOo/strongswan-5.3.5/src/libstrongswan/plugins/mysql/mysql_database.c:542
vpn charon[1986]: 05[LIB]
[0x7f14e2b6bd14]
vpn charon[1986]: 05[LIB] ->
/build/strongswan-UD5DOo/strongswan-5.3.5/src/libcharon/plugins/attr_sql/attr_sql_provider.c:93
vpn charon[1986]: 05[LIB]
[0x7f14e2b6bec1]
vpn charon[1986]: 05[LIB] ->
/build/strongswan-UD5DOo/strongswan-5.3.5/src/libcharon/plugins/attr_sql/attr_sql_provider.c:398
0x7f14f3b7f000 [0x7f14f3b93e74]
vpn charon[1986]: 05[LIB] ->
/build/strongswan-UD5DOo/strongswan-5.3.5/src/libstrongswan/collections/enumerator.c:438
0x7f14f36f6000 [0x7f14f373b35d]
vpn charon[1986]: 05[LIB] ->
/build/strongswan-UD5DOo/strongswan-5.3.5/src/libcharon/sa/ikev2/tasks/ike_config.c:400
0x7f14f36f6000 [0x7f14f372fb7f]
vpn charon[1986]: 05[LIB] ->
/build/strongswan-UD5DOo/strongswan-5.3.5/src/libcharon/sa/ikev2/task_manager_v2.c:781
0x7f14f36f6000 [0x7f14f3723ff7]
vpn charon[1986]: 05[LIB] ->
/build/strongswan-UD5DOo/strongswan-5.3.5/src/libcharon/sa/ike_sa.c:1402
0x7f14f36f6000 [0x7f14f371c981]
vpn charon[1986]: 05[LIB] ->
/build/strongswan-UD5DOo/strongswan-5.3.5/src/libcharon/processing/jobs/process_message_job.c:74
0x7f14f3b7f000 [0x7f14f3bacb3b]
vpn charon[1986]: 05[LIB] ->
/build/strongswan-UD5DOo/strongswan-5.3.5/src/libstrongswan/processing/processor.c:235
0x7f14f3b7f000 [0x7f14f3bbd89c]
vpn charon[1986]: 05[LIB] ->
/build/strongswan-UD5DOo/strongswan-5.3.5/src/libstrongswan/threading/thread.c:304
(discriminator 3)
0x7f14f34d9000 [0x7f14f34e06ba]
vpn charon[1986]: 05[LIB] -> ??:?
0x7f14f3110000 (clone+0x6d) [0x7f14f321682d]
vpn charon[1986]: 05[LIB] -> ??:?
vpn charon[1986]: 05[DMN] killing ourself, received critical signal
vpn ipsec_starter[32468]: charon has died -- restart scheduled (5sec)
Note that MySQL server is connected over the network, it's not on the
local machine if that's relevant.
--
Lauri Võsandi
tel: [phone]
blog: http://lauri.vosandi.com/