Introduction
cURL is a command-line tool and it stands for client URL. In a terminal (Linux) or command prompt (windows) it is generally written as “curl”. cURL is majorly utilized in doing all sorts of URL manipulations and transferring data while making various network protocols.
#Provides a detail description and flags for the command-line tool curl
curl --man
cURL provides a wide variety of flags for understanding and troubleshooting network calls to any particular URL. Some of the majorly used flags:
-v: verbose output which is an extremely useful option especially during troubleshooting/debugging in getting a detailed list of interaction between client and server. And, verbose can be done through layers. Flag -vv provides the second set of verbose which is way more detailed than -v.
--trace: trace is an ultimate troubleshooting flag it provided every detail on what curl is sending and receiving through the request and response from the server
-o: to save the output/stdout from curl to a file listed after the flag. -o flag is extensively used when a curl response from the server is a script that needs to be saved to a shell file in a local system.
-u: Pass authentication details for ex:
-u “username:password”
or
-u “apikey:{key}”
As our interests lie more on curl driven REST API request to RESTful services, we would limit the protocol to HTTP/HTTPS. And to understand what REST API calls are, we need to get an idea on API.
What is an API?
API is an abbreviated form of “Application Programming Interface”.An API helps with seamless connectivity. API allows two applications to talk to each other and exchange data in a secure manner.
Some of the most common examples of an APIs are - Google Maps , Facebook, Twitter, YouTube, LinkedIn and so on.
How do you identify, it’s an API?
There are many types of APIs. Let's just try to categorize them for better understanding;
- Java APIs, or interfaces within classes that let objects talk to each other in the Java programming language
- Web APIs - SOAP, RPC and the most popular is the REST
- And thousands of public and private organizational based APIs are available in the market.
In this blog, we will touch on RESTful APIs which are very common and widely known across Application Developers.
REST API
REST stands for “Representational State Transfer”. It is a set of rules and concepts that helps developers to model the application data as interrelated collections and objects. The resource representations in REST are self-descriptive. Fundamental operations on a REST API should be stateless, there should not be any stored context across the servers. And there should be a clear distinction between client and server where the state can be managed at the client end but not on a server. In return, this rule provides options for scalability, security, and also portability across data centers for the servers. The rules and concepts state that the REST API is broken down into a series of modules. For the developers, this level of Modularity provides flexibility. At the same time, it is a challenging feet to develop a REST endpoint from scratch.
An HTTP API protocol sent from a client to a REST endpoint is called a ‘request’ and the server provides the data-based on the REST endpoint model which in return is called a ‘response’. Applications are the ones that consume RESTful APIs, so the response doesn't have to be styled but it does provide pagination. Responses can be XML or JSON depending on the application. In some scenarios we can program the Target REST API endpoint to provide responses in either XML or JSON based on the input headers.
HTTP Methods for REST APIs
REST API’s suggest using a particular HTTP method on an API call from the client to the server. Below are standard HTTP Methods (frequently called as verbs):
● GET - Retrieve Resource representational information
● POST - Create a new sub resource
● PUT - Update Existing resource
● PATCH - Partial update on a resource
● DELETE - Delete a resource
These methods provide REST clients to perform four possible actions:
CRUD
C Create
R Read
U Update
D Delete
How do you make API calls using cURL?
Let's take an example of an existing REST API Endpoint JSON Placeholder (A simple HTTP Request & Response Service). JSON Placeholder
In this blog, we will cover GET, POST, PUT, PATCH, DELETE methods by using cURL.
Base URL: https://jsonplaceholder.typicode.com
Methods:
/GET
A simple GET method with media type in header and pagination:
Request:
curl -X GET "https://jsonplaceholder.typicode.com/posts?_page=1&_limit=2" \ -H "Accept: application/json
Response:
[
{
"userId": 1,
"id": 1,
"title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
"body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
},
{
"userId": 1,
"id": 2,
"title": "qui est esse",
"body": "est rerum tempore vitae\nsequi sint nihil reprehenderit dolor beatae ea dolores neque\nfugiat blanditiis voluptate porro vel nihil molestiae ut reiciendis\nqui aperiam non debitis possimus qui neque nisi nulla"
}
]
- -X GET: by default, curl considers the request method to be GET unless specified.
- -H: Header flag where the Key: accept and Value: application/json which indicates our request is expecting a response in JSON format.
- ?_page=1&_limit=2: pagination Page 1 with limit 2
To understand the request and response header and SSL handshakes curl provides a verbose flag:
Request:
curl -X GET "https://jsonplaceholder.typicode.com/posts?_page=1&_limit=2" \ -H "Accept: application/json" -v
Response:
Note: Unnecessary use of -X or --request, GET is already inferred.
* Trying 2606:4700:e6::ac40:c424...
* TCP_NODELAY set
* Connected to jsonplaceholder.typicode.com (2606:4700:e6::ac40:c424) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
* start date: Jul 29 00:00:00 2020 GMT
* expire date: Jul 29 12:00:00 2021 GMT
* subjectAltName: host "jsonplaceholder.typicode.com" matched cert's "*.typicode.com"
* issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fb070010800)
> GET /posts?_page=1&_limit=2 HTTP/2
> Host: jsonplaceholder.typicode.com
> User-Agent: curl/7.64.1
> Accept: application/json
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 200
< date: Fri, 07 Aug 2020 09:03:46 GMT
< content-type: application/json; charset=utf-8
< content-length: 600
< set-cookie: __cfduid=da9601881d1a1019d595f4e381f680d7e1596791026; expires=Sun, 06-Sep-20 09:03:46 GMT; path=/; domain=.typicode.com; HttpOnly; SameSite=Lax
< x-powered-by: Express
< x-ratelimit-limit: 1000
< x-ratelimit-remaining: 999
< x-ratelimit-reset: 1596790794
< vary: Origin, Accept-Encoding
< access-control-allow-credentials: true
< cache-control: max-age=43200
< pragma: no-cache
< expires: -1
< x-total-count: 100
< access-control-expose-headers: X-Total-Count, Link
< link: <http://jsonplaceholder.typicode.com/posts?_page=1&_limit=2>; rel="first", <http://jsonplaceholder.typicode.com/posts?_page=2&_limit=2>; rel="next", <http://jsonplaceholder.typicode.com/posts?_page=50&_limit=2>; rel="last"
< x-content-type-options: nosniff
< etag: W/"258-/AdFG/pwMUveUGKZ3vuwGAJYftA"
< via: 1.1 vegur
< cf-cache-status: HIT
< age: 290
< accept-ranges: bytes
< cf-request-id: 0469c28a480000f06092b24200000001
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< server: cloudflare
< cf-ray: 5befd38a0e64f060-EWR
[
{
"userId": 1,
"id": 1,
"title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
"body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
},
{
"userId": 1,
"id": 2,
"title": "qui est esse",
"body": "est rerum tempore vitae\nsequi sint nihil reprehenderit dolor beatae ea dolores neque\nfugiat blanditiis voluptate porro vel nihil molestiae ut reiciendis\nqui aperiam non debitis possimus qui neque nisi nulla"
}
* Connection #0 to host jsonplaceholder.typicode.com left intact
]* Closing connection 0
The above is the detail list of transport layer communication between the client and server
/POST
A simple POST method with media type in header and json request body:
Request:
curl -X POST 'https://jsonplaceholder.typicode.com/posts' \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' -d '{
"userId": 10,
"title": "Just Infinity",
"body": "Infinity has no beginning, which has no beginning there is no end"
}'
Response:
{
"userId": 10,
"title": "Just Infinity",
"body": "Infinity has no beginning, which has no beginning there is no end",
"id": 101
}
- -H 'Accept: application/json’: This header Parameter passing the media type application JSON which indicates the client accepts JSON body as a response
- -H 'Content-Type: application/json': This header parameter passing the media type application JSON is indicating the post request the body is in JSON format
- -d: indicates the short version of data
As you can see, from the response the POST request had created a new resource with id:101. In case you send a request body the same as the existing resource it would still create a new resource with a new ID as the POST method is not idempotent.
/PUT
A simple PUT method with media type in header and json request body:
Request:
curl -X PUT 'https://jsonplaceholder.typicode.com/posts/10' \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' -d '{
"userId": 10,
"title": "Just Infinity",
"body": "Infinity has no beginning, which has no beginning there is no end"
}'
Response:
{
"userId": 10,
"title": "Just Infinity",
"body": "Infinity has no beginning, which has no beginning there is no end",
"id": 10
}
- /10 in the URL indicates we are updating the existing resource id:10.
As you can see, PUT is an idempotent, and it would be updating the current existing resource. And in case any JSON objects are missing in your request body it would be lost permanently as the current resource gets updated with the new request body sent.
For example, from the above request body, we eliminate the JSON object “body”.
Then:
Request:
curl -X PUT 'https://jsonplaceholder.typicode.com/posts/10' \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' -d '{ "userId": 10, "title": "Just Infinity" }'
Response:
{
"userId": 10,
"title": "Just Infinity",
"id": 10
}
/PATCH
A simple PATCH method with media type in header and json request body:
Request:
curl -X PATCH 'https://jsonplaceholder.typicode.com/posts/10' \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' -d '{
"title": "Just Infinity"
}'
Response:
{
"userId": 1,
"id": 10,
"title": "Just Infinity",
"body": "quo et expedita modi cum officia vel magni\ndoloribus qui repudiandae\nvero nisi sit\nquos veniam quod sed accusamus veritatis error"
}
As you can see, PATCH does not need the whole body to be sent as part of the request. One can limit that to the single JSON object. And only the parameter that is sent would be updated.
/DELETE
A simple DELETE method with media type in header and json request body:
Request:
curl -X DELETE 'https://jsonplaceholder.typicode.com/posts/10'
Response:
{}
As you can see, DELETE simply removes the existing resource in our case resource 10 was removed.
Conclusion
To put this in nutshell, cURL is really proving a command-line helper to developers when working with APIs and providing immense support for making requests, connecting with APIs, troubleshooting the issues using verbose, multiple security options to access the APIs securely and many more. I hope this blog can give the basic understanding what is cURL, how to construct a cURL request, get the response and some important flags to troubleshoot while working with APIs.
In the future, will be coming up with another blog “cURL for Apigee API proxy” which will be mainly focused on create, build, deploy proxy on Apigee edge and use cURL to test the proxy passing “Bearer” as Authorization header to support OAuth2 client credentials.