In a previous post I discussed creating an HTTP API in q. Back then I mentioned that I may explore adding HTTPS support at a later point; I’ve recently had a chance to give this a try, and thanks to Let’s Encrypt it’s very easy to do!
This will work regardless of whether you are using the
webapi library from my
previous post (now available from conda:
conda install -c jmcmurray webapi -
see this post
about q package management for more details on that) or another custom
.z.pp definition (or even the default q HTTP interface).
Step one is to generate your certificates; this is pretty simple using certbot from Let’s Encrypt. This is a fairly straightforward process, documented here [instructions for various OSes available]. For example on Ubuntu, once certbot is installed we simply do something like:
sudo certbot certonly --standalone -d jmcmurray.hopto.org
jmcmurray.hopto.org is a domain I control). Certs will be
generated in e.g.
jonny@grizzly ~ $ sudo ls -lFh /etc/letsencrypt/live/jmcmurray.hopto.org total 4.0K lrwxrwxrwx 1 root root 43 Oct 25 15:08 cert.pem -> ../../archive/jmcmurray.hopto.org/cert1.pem lrwxrwxrwx 1 root root 44 Oct 25 15:08 chain.pem -> ../../archive/jmcmurray.hopto.org/chain1.pem lrwxrwxrwx 1 root root 48 Oct 25 15:08 fullchain.pem -> ../../archive/jmcmurray.hopto.org/fullchain1.pem lrwxrwxrwx 1 root root 41 Dec 8 14:10 jmcmurray.hopto.org -> /etc/letsencrypt/live/jmcmurray.hopto.org/ lrwxrwxrwx 1 root root 46 Oct 25 15:08 privkey.pem -> ../../archive/jmcmurray.hopto.org/privkey1.pem -rw-r--r-- 1 root root 682 Oct 25 15:08 README
As the certs are created as root, we have a couple of options. We could run our q server as root, allowing us to read the certs. Alternatively, we can copy (as root) to another readable location e.g.
jonny@grizzly ~ $ mkdir certs jonny@grizzly ~ $ sudo su root@grizzly:/home/jonny# cp /etc/letsencrypt/live/jmcmurray.hopto.org/* certs/ cp: -r not specified; omitting directory '/etc/letsencrypt/live/jmcmurray.hopto.org/jmcmurray.hopto.org' root@grizzly:/home/jonny# exit exit jonny@grizzly ~ $ ll certs/ total 20K -rw-r--r-- 1 root root 2.2K Dec 8 15:06 cert.pem -rw-r--r-- 1 root root 1.7K Dec 8 15:06 chain.pem -rw-r--r-- 1 root root 3.8K Dec 8 15:06 fullchain.pem -rw-r--r-- 1 root root 1.7K Dec 8 15:06 privkey.pem -rw-r--r-- 1 root root 682 Dec 8 15:06 README
Now we can set the necessary environment variables & load up our q session:
jonny@grizzly ~ $ export SSL_CERT_FILE=~/certs/fullchain.pem jonny@grizzly ~ $ export SSL_KEY_FILE=~/certs/privkey.pem jonny@grizzly ~ $ q -E 1 -p 8100 KDB+ 3.6 2018.05.17 Copyright (C) 1993-2018 Kx Systems l32/ 2()core 1944MB jonny grizzly 127.0.1.1 NONEXPIRE q).utl.require"webapi" q)\l git/qwebapi/example.q
(Here I load my
webapi module previously mentioned & the same example script
I used in my previous post on HTTP APIs)
Now, we can query the API over HTTPS:
All in all, the process is pretty simple & straightforward, and takes about 5 minutes. As a couple of closing notes:
-E 1as I showed here still allows HTTP access; with
-E 2only HTTPS is allowed (however, HTTP requests will not be redirected to HTTPS - I’ll be looking into this in future)
- As per the kx docs
-u 1should be used to prevent remote access to your server key file, & q should not be run from a directory where the keys can be accessed
UPDATE: After publishing this post, I realised that kdb+ seemingly does not publish the full certificate chain it is provided, only the server certificate.
This means that, when using Let’s Encrypt, the intermediate LE certificate is not provided to clients. Some clients (e.g. web browsers) will download the intermediate(s) as necessary & the user will likely not notice anything is “wrong”. Other clients (e.g. another q session) will not, and will therefore fail to verify the server certificate.
Report from SSL Labs
In order to send an HTTPS query from another q session, you will need to either
disable SSL server verification (set env var
starting q) or add your server cert to the CA bundle you are using.