Where does libcurl3 dependancy come from?

Linux
1

Hi all, we build Waveform on an Ubuntu 16.04 machine and it seems that in doing this it builds in a dependancy to libcurl3.

Ubuntu 18.04, the new LTS only has libcurl4 which means Waveform no longer works on these or Debian 9 based systems.

Do any of the JUCE team know where this gets pulled in from? In the makefile all I can see is libcurl which I would have thought would just use whatever is installed on the system.
Is it perhaps from the webkit2gtk-4.0 gtk+-x11-3.0 dependancies?

If I remove them (we don’t use the web browser on Linux), will the app be happy to use libcurl4? Or is this simply coming from the fact that it’s being built on 16.04?

There was some discussion on KVR here, I’m just trying to ascertain where the problem is arising from before spending a lot of time fixing it and preparing new builds.

Thanks for any help.

2

I think it’s coming from the fact that it is being built on 16.04 and has nothing to do with webkit2gtk-4.0 or gtk+-x11-3.0 as you are linking these in as shared libraries - the dependencies of these libraries don’t really bother you.

More specifically, it depends on which libcurl-dev package you install.

As you probably know there are always two types of packages for (most) shared libraries. A runtime package and a devel package.

The runtime package only contains the shared library which will include the api version suffix: i.e. libcurl4 etc.

The devel package contains all the headers you need, usually a static library variant, a pkg-config config file, and then a symlink (libcurl -> libcurl4) which will allow you to just link against library name without the version (-lcurl) - this is what JUCE does.

I think for libcurl you have the choice of different versions of libcurl-dev packages (libcurl3-dev, libcurl4-dev etc.), but I’m not sure if libcurl4-dev is already available in Ubuntu 16.04 - and if it is, if the runtime package will be installed by default in Ubuntu 16.04. If it isn’t your users will need to install it themselves (or you will need to add a dependency to your debian package).

3

I’ve also encountered this libcurl issue. I’m not deeply familiar with Linux though. That’s why this post is quite verbose…

Compile a standalone application with JUCE_USE_CURL enabled in Ubuntu 16.04. Try to launch it in Ubuntu 18.04. It won’t launch, you’ll get

/usr/lib/x86_64-linux-gnu/libcurl.so.4: version `CURL_OPENSSL_3' not found

Do it the other way around and it won’t launch either:

/usr/lib/x86_64-linux-gnu/libcurl.so.4: version `CURL_OPENSSL_4' not found

Under Ubuntu 16.04, the latest available libcurl is version 4.4.0
(apt policy libcurl4-openssl-dev -> 7.47.0-1ubuntu2.8):

/usr/lib/x86_64-linux-gnu$ ll libcurl*
-rw-r--r-- 1 root root 947448 Mai  9 15:12 libcurl.a
lrwxrwxrwx 1 root root     19 Mai  9 15:12 libcurl-gnutls.so.3 -> libcurl-gnutls.so.4
lrwxrwxrwx 1 root root     23 Mai  9 15:12 libcurl-gnutls.so.4 -> libcurl-gnutls.so.4.4.0
-rw-r--r-- 1 root root 444800 Mai  9 15:12 libcurl-gnutls.so.4.4.0
-rw-r--r-- 1 root root    953 Mai  9 15:11 libcurl.la
lrwxrwxrwx 1 root root     16 Mai  9 15:12 libcurl.so -> libcurl.so.4.4.0
lrwxrwxrwx 1 root root     12 Mai  9 15:12 libcurl.so.3 -> libcurl.so.4
lrwxrwxrwx 1 root root     16 Mai  9 15:12 libcurl.so.4 -> libcurl.so.4.4.0
-rw-r--r-- 1 root root 452992 Mai  9 15:12 libcurl.so.4.4.0

Under Ubuntu 18.04, the latest available libcurl is version 4.5.0
(apt policy libcurl4-openssl-dev -> 7.58.0-2ubuntu3.1):

/usr/lib/x86_64-linux-gnu$ ll libcurl*
-rw-r--r-- 1 root root 1057664 Mai  8 19:47 libcurl.a
lrwxrwxrwx 1 root root      19 Mai  8 19:47 libcurl-gnutls.so.3 -> libcurl-gnutls.so.4
lrwxrwxrwx 1 root root      23 Mai  8 19:47 libcurl-gnutls.so.4 -> libcurl-gnutls.so.4.5.0
-rw-r--r-- 1 root root  510408 Mai  8 19:47 libcurl-gnutls.so.4.5.0
-rw-r--r-- 1 root root     951 Mai  8 19:47 libcurl.la
lrwxrwxrwx 1 root root      16 Mai  8 19:47 libcurl.so -> libcurl.so.4.5.0
lrwxrwxrwx 1 root root      16 Mai  8 19:47 libcurl.so.4 -> libcurl.so.4.5.0
-rw-r--r-- 1 root root  518600 Mai  8 19:47 libcurl.so.4.5.0

Where does the symbol CURL_OPENSSL_3/4 come from?

“That symbol looks like something Debian/Ubuntu have introduced. There is no such name (CURL_OPENSSL_3) in any curl source code or build script that comes from us.” (https://github.com/curl/curl/issues/2433)

One extreme solution that works… and most likely breaks a lot of other applications: Downgrade libcurl in Ubuntu 18.04

  • Add the 16.04 repository “deb http://archive.ubuntu.com/ubuntu xenial-updates universe main” to the /etc/apt/sources.list in 18.04.
  • sudo apt install libcurl3=7.47.0-1ubuntu2.8 libcurl4-openssl-dev=7.47.0-1ubuntu2.8 -t xenial-updates

@fabian, @dave96 Any idea of a proper way to deal with this (with a closed source application)? IIRC libraries are not meant to be bundled with binaries under Linux.

Does someone have experience with JUCE and https://snapcraft.io? Probably not going to work with an audio plugin…

4

We used to build a static version of curl when we distributed the Projucer. It’s a bit fiddly though: you need to build it from source and disable most of the protocols (except http and https) (have a look at ./configure --help). You’ll also need to add configure flags to actually build a static version of the lib (I think it was --disable-shared --enable-static).

Also make sure to add --prefix=/some/directory/in/my/home/directory so that the libraries don’t get install onto your system. Once everything is installed you’ll need to set the pkg-config environment variable PKG_CONFIG_PATH to point to the pkgconfig folder that curl created in your installation directory.

If you’ve got everything working then pkg-config --libs --static libcurl should print out the libraries you need.

I think you might also need to edit the Makefile that the Projucer creates to add the --static to the pkg-config invocation.

5

I’ve started looking into statically linking libcurl under Linux this morning. Linking with

-static-libstdc++ 
-static-libgcc 
-Wl,-Bstatic -lcurl -Wl,-Bdynamic

makes it actually possible to launch a binary in Ubuntu 18.04 that was compiled under Ubuntu 16.04.

Now the next step is to build curl with the minimal feature set needed (and minimal dependencies). @fabian Thank you for your tips!

6

That’s great Samuel, I spent a bit of time looking in to this on Saturday and reached the conclusion that it may just be best to build a separate version for Ubuntu 18.04 and depreciate the 16.04 version with our next major release. However, I ran in to problems with debbuild on the newer Linux version and this would also mean having to run two Linux build servers, not ideal.

If you get the static linking working let us know. Cheers! Dave

7

I resign - for now.

I was able to build a static libcurl with ssl, but when my application tries to communicate with a website (https), it doesn’t work. It might only be a matter of finding the right configure options for libcurl.

Maybe someone is interested in what I’ve done:

Download the libcurl source from
https://curl.haxx.se/download.html
https://curl.haxx.se/docs/install.html

In the source folder:
env PKG_CONFIG_PATH=/usr/lib/x86_64-linux-gnu/pkgconfig/ ./configure --prefix=$HOME/lib --disable-ares --disable-shared --enable-static --enable-http --disable-ftp --disable-file --disable-ldap --disable-ldaps --disable-rtsp --disable-proxy --disable-dict --disable-telnet --disable-tftp --disable-pop3 --disable-imap --disable-smb --disable-smtp --disable-gopher --disable-manual --enable-libcurl-option --enable-ipv6 --enable-pthreads --disable-verbose --disable-sspi --disable-crypto-auth --disable-ntlm-wb --disable-tls-srp --disable-unix-sockets --disable-cookies --without-zlib --without-brotli --with-ssl --without-libpsl --without-libmetalink --without-librtmp --without-libidn2 --with-nghttp2 --without-zsh-functions-dir

This summary is shown

configure: Configured to build curl/libcurl:

curl version:     7.60.0
Host setup:       x86_64-pc-linux-gnu
Install prefix:   /home/sam/lib
Compiler:         gcc
SSL support:      enabled (OpenSSL)
SSH support:      no      (--with-libssh2)
zlib support:     no      (--with-zlib)
brotli support:   no      (--with-brotli)
GSS-API support:  no      (--with-gssapi)
TLS-SRP support:  no      (--enable-tls-srp)
resolver:         POSIX threaded
IPv6 support:     enabled
Unix sockets support: no      (--enable-unix-sockets)
IDN support:      no      (--with-{libidn2,winidn})
Build libcurl:    Shared=no, Static=yes
Built-in manual:  no      (--enable-manual)
--libcurl option: enabled (--disable-libcurl-option)
Verbose errors:   no
SSPI support:     no      (--enable-sspi)
ca cert bundle:   /etc/ssl/certs/ca-certificates.crt
ca cert path:     no
ca fallback:      no
LDAP support:     no      (--enable-ldap / --with-ldap-lib / --with-lber-lib)
LDAPS support:    no      (--enable-ldaps)
RTSP support:     no      (--enable-rtsp)
RTMP support:     no      (--with-librtmp)
metalink support: no      (--with-libmetalink)
PSL support:      no      (--with-libpsl)
HTTP2 support:    disabled (--with-nghttp2)
Protocols:        HTTP HTTPS

Build:

make
make install

Verify:

env PKG_CONFIG_PATH=/home/sam/lib/lib/pkgconfig pkg-config --libs --static libcurl

Edit the Makefile of the JUCE application. For the CONFIG=Release, add “–static” to the pkg-config:

    JUCE_LDFLAGS += $(TARGET_ARCH) -L$(JUCE_BINDIR) -L$(JUCE_LIBDIR) -L/usr/X11R6/lib/ $(shell pkg-config --libs --static alsa freetype2 libcurl x11 xext xinerama webkit2gtk-4.0 gtk+-x11-3.0) -fvisibility=hidden -static-libstdc++  -static-libgcc  -Wl,-Bstatic -lcurl -Wl,-Bdynamic -ldl -lpthread -lrt $(LDFLAGS)

Add some additional libraries:

sudo apt install liblzma-dev libsystemd-dev libselinux1-dev libffi-dev

Compile the JUCE application:

env PKG_CONFIG_PATH=/home/sam/lib/lib/pkgconfig make CONFIG=Release

A helpful blog entry: https://insanecoding.blogspot.com/2012/07/creating-portable-linux-binaries.html

8

What exactly doesn’t work? What error are you getting? If you are getting SSL verification errors then you are linking curl incorrectly. You need to link curl itself as a static library, but any SSL libraries (like openSSL or gnutls) must always be dynamic for security reasons. What did pkg-config tell you which libs need to be linked:

env PKG_CONFIG_PATH=/home/sam/lib/lib/pkgconfig pkg-config --libs --static libcurl

9 
-L/home/sam/lib/lib -lcurl -lssl -lcrypto -lssl -lcrypto

No error. My network monitor shows that a connection to the server gets established. According to the server log the communication from the application to the server works. But the juce_product_unlocking (that’s the reason I need ssl) doesn’t work, “Unexpected or corrupted reply from…”. So the UnlockResult doesn’t contain a result nor an error message. I can’t work with the debugger, only the release build compiles. Linking of the debug build fails:

env PKG_CONFIG_PATH=/home/sam/lib/lib/pkgconfig make AR=gcc-ar CONFIG=Debug
Linking Myapp - App
/home/sam/lib/lib/libcurl.a(libcurl_la-openssl.o): In function `Curl_ossl_sha256sum':
openssl.c:(.text+0x11c): undefined reference to `EVP_MD_CTX_create'
openssl.c:(.text+0x124): undefined reference to `EVP_sha256'
openssl.c:(.text+0x131): undefined reference to `EVP_DigestInit_ex'
...

libssl-dbg is installed. Do I also need to manually compile openssl in order to be able to compile a debug build?

10

It would really be good for you to somehow get a debug build working. It could be that your error has nothing to do with the way you link anymore - could be some completely unrelated undefined behaviour in your app.

If you are using pkg-config --static shouldn’t the result also be in the static linking group? I try to put as much as possible in that group. I think I only leave libc, librt and libdl in the dynamic group.

Also do you really need webkit? Are you using the juce’s WebBrowserComponent? If not then you should disable that in the Projucer (click on the juce_gui_extra module) as that adds a lot of dependencies. You also will not need gtk+-x11-3.0 then.

Also, if you specify linking to libpthread with the compiler option -pthread instead of -lpthread then gcc will also link that statically.

I haven’t done this in a while but I think my JUCE_LDFLAGS looked something like this (without WebBrowserComponent):

JUCE_LDFLAGS += $(TARGET_ARCH) -L$(JUCE_BINDIR) -L$(JUCE_LIBDIR) -L/usr/X11R6/lib/ -static-libstdc++  -static-libgcc -pthread -Wl,-Bstatic $(shell pkg-config --libs --static alsa freetype2 libcurl x11 xext xinerama ) -fvisibility=hidden -Wl,-Bdynamic -ldl -lrt $(LDFLAGS)
11

Thanks again for your help @fabian!

The procedure described in my previous post actually works, a server side issue caused the failure. So, compiling in Ubuntu 16 with a statically linked libcurl and running the binary in Ubuntu 18 actually works!

But it wont run in Debian (9.4) and maybe in no other distribution than Ubuntu. The problem:

/usr/lib/x86_64-linux-gnu/libssl.so.1.0.2: version `OPENSSL_1.0.2' not found

Maybe related to the Ubuntu specific CURL_OPENSSL_3 I’ve mentioned earlier.

For my current project, I resign and wont use curl at all. Without curl, the binary created in Ubuntu 16 also works in Debian 9.

12

Sorry to come back to this after a while, I didn’t want to fuss around with the static linking as the steps above look… tricky…

I finally managed to get a build working on Ubuntu 18.04 after spending ages debugging incompatibility problems with debuild, only to discover that this problem is also present in all of our plugins too! (Obviously)

I really can’t face maintaining two separate builds for all of our products for pre and post 18.04 (this would also mean building and maintaining another build server).

It looks like the static linking option above isn’t even a silver bullet as it doesn’t work with non-Ubuntu versions of Debian?

Does anyone have any better ideas? Are there any alternatives to curl? We use it for authorisation and file downloading in places (through juce::URL).

13

Rather than having a hard dependancy, would using DynamicLibrary to load only the functions required by juce_curl_Network.cpp work?

Could this bypass any hard dependancies on a curl version? Is there anything version specific in <curl/curl.h>?

1 Like
14

So what is the official recommendation on this issue in 2021? I read another thread that recommended compiling against an older version of Ubuntu to make things more compatible. I ended up compiling in Ubuntu 16.04. I just got my first user feedback email saying:

I just tried to install, but with no success. it reports:
./MyAppName: /lib/x86_64-linux-gnu/libcurl.so.4: version `CURL_OPENSSL_3’ not found (required by ./MyAppName)

Thanks!

15

We use JUCE_LOAD_CURL_SYMBOLS_LAZILY=1 to load curl dynamically which should work with both curl versions.

2 Likes
16

Aha! Thanks. I’ll give this a try. :slight_smile: