Image tagging and finding made easy with AI
This post is part of a series on the features, knowledge and insight that came out of our “Practitioner Love” hackathon in mid-February. What do we mean by “Practitioner”? If you’re a writer, editor, marketer content creator or non-technical user of Contentful — that’s you!
Organizing a collection of thousands of images can be daunting and time-intensive. If you or your team members haven’t taken the time to organize your images by tagging, adding detailed titles and meta-descriptions, that disorder can eat up your time. Every new blog post turns into a game of hide-and-seek as you scroll and scroll, trying to find your feature image, or that infographic your designer dropped into the library.
But it doesn’t have to be this way. Organizing, and finding your images, can take seconds, making your content operations workflow a whole lot smoother.
The need for consistent naming systems
With Contentful, all media is organized under a single section of the web app. It’s a whole lot easier than searching multiple CMSes for your content; everything is curated and stored in a single authoring hub, ready to be used and distributed to any platform.
But even the most straightforward storage solution can become messy when teams don’t have standardized or consistent naming systems, or don’t describe images in the required fields. Ever uploaded a picture for a blog post and filled in the necessary fields with nonsense or a quick, but incomplete, description? Be honest, of course you have!
This gets more complex when image metadata is needed to enable SEO or accessibility. It’s why Contentful recommends wrapping images in a custom content type, as this allows full customization of metadata fields based on the project. When you follow this approach, two representations of each image exist, and proper library organization becomes vital.
Enter Amazon AWS Rekognition
The fastest, easiest way to make this happen is to hook up with some of the smartest AI technology on the market. Amazon AWS Rekognition identifies objects, people, text, scenes and activities in images and videos. It also provides highly accurate facial analysis and facial recognition. This makes images searchable and automatically creates metadata for your videos. Not only does this AI tool make your content libraries searchable, it also makes laborious tasks such as image tagging and moderation a breeze. You can read more about Amazon AWS Rekognition here.
AI goes even deeper than simply being able to search your images. Based on deep learning technology developed by Amazon’s data scientists, Amazon Rekognition is constantly learning from new data collected by Amazon’s data engineers. This frees users from the challenging task of having to build and maintain their own machine learning model and data set. Rekognition machine learning functionality is available as an API-based service, making it a perfect candidate for integration with Contentful.
We decided this was something our customers needed access to. We manage our own content libraries and are empathetic to how much time this task can drain from a work day. As part of our “Practitioner Love” hackathon, a taskforce got together and focused on making an AWS Rekognition UI extension available for all users. The result of all that hard work is now available as a UI extension on the marketplace. It can be installed in a space from here.
The extension can be added to the sidebar of any content type that includes both an image media field and a field to hold tags. After adding an image to an entry by clicking ‘Auto-tag image,’ the image passes through Amazon Rekognition and descriptive tags get added to the entry. This enables the (nearly) effortless curation of large image libraries and tag standardization across multiple authors in the same space.
Extending Contentful with image tagging AI
Contentful is highly extensible. It’s one of the best things about our platform. Contentful’s extensibility makes it easy to integrate additional services such as AI to enrich content. This can free up valuable authoring time to focus on the creative parts of content production and leave the boring stuff to the machine. Contentful offers two ways of adding new functionality to content: webhooks and UI Extensions.
Webhooks automatically trigger custom functionality when content is added or changed. UI extensions are a little different — they allow for customization of the actual content editing experience by adding different editor field design and sidebar widgets. We chose to build automated image tagging as a UI extension. Why? Because it gives the content creator full control over the final outcome. This way, tags can be generated after an image is added, and then further curated to whatever the use case requires.
The design of the UI extension is simple; it’s just an add-on button that triggers the automated tagging process. To give you further control, we also added the possibility to keep existing tags.
Once the ‘Auto-tag image’ button is clicked, the extension automatically retrieves the underlying asset containing the image file. The image is then sent to Rekognition’s labeling service, which assigns it a set of tags and corresponding confidence levels. The confidence levels enable the extension to only focus on adding tags that it knows will match the image content. Tags are then added to the tag field of the wrapper content type.
Going even further with AI and Contentful
Automated tagging of images is just one use case that can be automated by integrating third-party AI services. For example, one customer integrated Contentful with IBM Watson for sentiment analysis (using Natural Language Understanding). The options are growing by the day and include AI-based text analysis tools such as automated translation, grammar checks and SEO analysis. Integrating these tools can greatly enhance the authoring experience and shorten publishing time while reducing critical errors.
Sure, checking for wayward commas and hyphens is awesome, but plagiarism and fraud prevention AI tools are also gaining popularity. Major machine-learning providers are offering services to analyze and evaluate content.
The capabilities of machine learning are growing steadily, covering ever more application fields. Contentful’s flexible, extensible platform provides the perfect foundation for exploring these different AI options. Automatic image tagging is just the beginning –– now what are you going to do with all those hours you just got back?
