How to quickly access API responses with curl and jq
Productivity plays a significant role in the technology-driven world we’re living in. I still remember when I started programming seeing the senior devs being all about shortcuts and little helper programs. It goes from tweaking the terminal with aliases for often-used commands over cronjobs doing daily tasks to apps the whole purpose of which is to save time. There are even people pushing it to the next level by automating coffee machines, answering support emails or sending messages to their partner. It was (and still is) all about ways to do more with less.
I use a lot of APIs these days. It doesn’t matter what problem I have there seems always to be a pre-made cloud service fitting my needs.
I want to have a CLI tool for everything
Not counting a few prototypes I built a mobile app and my personal website with Contentful. Using the API works great, but the development process can get tedious at times because you have to make a request to see what data your application uses.
A lot of people use tools like Postman to fire requests and to quickly access API data. I was never a big fan of this approach. Don’t get me wrong – tools like Postman are great, but I don’t need another program running when I’m developing. I want an easy and quick(!) solution right in my terminal because it runs continuously and is perfectly tailored to my needs.
I checked the Contentful ecosystem looking for a CLI tool to use. There is none so far, so I started writing a little Node.js module to help me out. Soon I got the feeling that I’m probably over-engineering.
This should be easily doable with shell scripting
Shell scripting is one of these things that I’m not doing very often, and it still scares me sometimes. Nevertheless, I decided to give it a shot.
What I was aiming for was a simple ctf command which helps me to access quickly data of a particular resource stored in Contentful. In Contentful you have the option to set up several “data buckets” which we call "spaces".
I started setting up a function which should fire a curl request to an HTTP endpoint.
So far so good – but most APIs require you to authenticate your requests before consuming resources. The authentication usually happens using an access token. I could have hardcoded this token in my dotfiles now, but these are on GitHub, and I surely don’t want to expose any sensitive information in a public repository.
With the hardcoding option off the list, I went for a different tack. A lot of CLI tools place hidden files in your home directory, so I decided to do the same for my helper function. I created the hidden file .contentful.json and added configurations for different spaces.
How to read JSON files?
With the config file in place, it only has to be read and parsed.
To be kind to my future self, the first thing I did was to implement a presence check of the config file. The whole purpose of dotfiles is to be able to set up a new environment quickly when switching machines. Configuration files are usually not included in the dotfiles, so I wanted to give me a bit of guidance for the future.
At this point, the only thing missing is parsing the JSON file. In Node.js this is a no-brainer, but I didn’t know how to do this within a shell environment. It turns out there is a module called jq available. You can install it easily via brew and be ready to go. I added it to the installation section of my dotfiles and started parsing my config file.
jq enables you to define a JSON path for the data you want to access, a path to a JSON file, and then filters the JSON for you. In this case, I was interested in the website property.
Piping data into jq is also possible. 🎉
The -r flag helps to remove quotes from accessed string values.
With these two ways to execture jq with JSON data, now I can easily parse it and extract the config values I need to make the request against the Contentful API.
And now for the cool part. After storing the required information in variables, all I need to do is to place them inside of the curl command and to access data quickly via the new ctf command.
Shell scripting for productivity!
Shell scripting is powerful and once you get your head around it can be extremely useful. You see the first version of my function below:
This function is 15 lines of code in my shell config! I iterated over the first version already several times to include more API endpoints, and without realizing it, I’m already heavily relying on this CLI functionality when working against the Contentful API. And these lines are quickly adjusted to fire against any API out there, too. I'm sure I'll enrich my terminal with more functionality in the near future – creating calendar appointments is immediately coming to my mind.
Thank God I didn't write a Node.js module for that! Writing a shell script saved me a lot of time so that I could spend my time on important things. And this is what productivity is about, or?
