Handy Curl Commands

Curl is a super powerful tool that can be used for testing and scripting. It is probably one of my favorite tools and I use it on a daily basis in my work.
Posted below are some commands that I use for troubleshooting and they barely even scratch the surface of what this tool can do.

Pass a host header to an IP address to test a virtual host:
-I (Dumps HTTP response headers)
-k (ignores SSL errors)
-H (Passes host headers)

1
2
3
4
5
6
7
8
9
$ curl  -IkH 'Host: www.wafmaster.com' https://72.249.84.27
HTTP/1.1 200 OK
Date: Mon, 03 Sep 2018 11:45:37 GMT
Content-Type: text/html
Cache-Control: no-cache, no-store, must-revalidate
Pragma: no-cache
Expires: 0
Content-Length: 1668
Connection: keep-alive

Set a user agent on the request header:
-A (Set user agent)
-I (Dumps HTTP response headers)

1
2
3
4
5
6
7
8
9
$ curl  -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36" -I https://www.wafmaster.com
HTTP/1.1 200 OK
Date: Mon, 03 Sep 2018 11:45:53 GMT
Content-Type: text/html
Cache-Control: no-cache, no-store, must-revalidate
Pragma: no-cache
Expires: 0
Content-Length: 1668
Connection: keep-alive

Test initial GET response time:
-s (Silent)
grep -v (Nix based OS excludes the HTML tags)

1
2
3
4
5
$ time curl -s  https://www.wafmaster.com | grep -v "<"
 
real	0m0.338s
user	0m0.019s
sys	0m0.007s

Run a verbose curl for extra information:
–verbose (Verbose mode)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
$ curl --verbose https://www.wafmaster.com
* Rebuilt URL to: https://www.wafmaster.com/
*   Trying 72.249.84.27...
* TCP_NODELAY set
* Connected to www.wafmaster.com (72.249.84.27) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=wafmaster.com
*  start date: Jul 14 10:23:56 2018 GMT
*  expire date: Oct 12 10:23:56 2018 GMT
*  subjectAltName: host "www.wafmaster.com" matched cert's "www.wafmaster.com"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: www.wafmaster.com
> User-Agent: curl/7.54.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Mon, 03 Sep 2018 11:59:41 GMT
< Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips PHP/5.4.16
< Last-Modified: Fri, 13 Jul 2018 10:16:12 GMT
< ETag: "eb-570dec4ee3fbe"
< Accept-Ranges: bytes
< Content-Length: 235
< Content-Type: text/html; charset=UTF-8
< 
<html>
<body bgcolor="black">
<center>
<p>Hello Nerd</p>
</center>
</html>
* Connection #0 to host www.wafmaster.com left intact

Display the version of curl you are using:
-V (version)

1
2
3
4
$ curl -V
curl 7.54.0 (x86_64-apple-darwin17.0) libcurl/7.54.0 LibreSSL/2.0.20 zlib/1.2.11 nghttp2/1.24.0
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz HTTP2 UnixSockets HTTPS-proxy

Nginx Reverse Proxy Example

In a previous post I shared a virtual host configuration for an Apache reverse proxy and thought I would do the same for Nginx.

Nginx is a very light weight and effective web server and reverse proxy!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
 server {
 
    listen 443;
    listen  [::]:443;
    server_name www.somedomain.com somedomain.com;
 
    ssl_certificate           /etc/ssl/certs/cert.pem;
    ssl_certificate_key       /etc/ssl/certs/key.pem;
 
    ssl on;
    ssl_session_cache  builtin:1000  shared:SSL:10m;
    ssl_protocols  TLSv1.1 TLSv1.2;
    ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
    ssl_prefer_server_ciphers on;
 
    access_log            /var/log/nginx/somedomain.com.access.log;
 
    proxy_set_header HOST $host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Custom-XFF $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 
 location / {
      # Fix the “It appears that your reverse proxy set up is broken" error.
      proxy_pass          https://10.0.1.10:443;
      proxy_read_timeout  90;
 
    }
  }