Extending on the Azure Cognitive Services from day 5’s challenge to encompass Computer Vision, Day 15’s challenge is to take a given image and return a description of the image and keywords to describe object(s) in the image.

Constructing a Solution

In this case, I’m assuming an incoming image URL for assistive analysis – where an image description and keywords will be returned. For this, I’ve created an object/class with those 3 attributes as well as a constructor to create the object with the incoming image URL.

class ImageInfo {
    public imgUrl: string;
    public description: string;
    public keywords: [string];

    constructor(imgUrl:string) {
        this.imgUrl = imgUrl;
    }
}

This ImageInfo object is one of the parameters for a utility function that accesses the Azure Computer Vision API via a POST request. The REST request is called using request-promise-native.

The request parameter for visualFeatures accepts multiple comma-separated options – for this example, I’d like Objects and Descriptions. Before exiting, the function sets the ImageInfo object attributes for the description and keywords.

async function imageDetection(language_endpoint: string, subscription_key: string, imageInfo: ImageInfo) {

    const params = {
        'visualFeatures': 'Objects,Description',
        'details': '',
        'language': 'en'
    };

    var options = {
        uri: language_endpoint,
        qs: params,
        body: '{"url": ' + '"' + imageInfo.imgUrl + '"}',
        headers: {
            'Content-Type': 'application/json',
            'Ocp-Apim-Subscription-Key' : subscription_key
        }
    };

    var imgResult = await rpn.post(options);
    console.log(imgResult);
    
    imageInfo.description = JSON.parse(imgResult).description.captions[0].text;
    imageInfo.keywords = JSON.parse(imgResult).description.tags;

}

Wrap the imageDetection() function up with some housekeeping:

• accessing environment variables for the subscription key and endpoint URL
• pull the image URL from the calling request’s body
• wrap the imageDetection() function in a try…catch to allow returning either the completed ImageInfo object or an error message to the requestor

Complete Solution

https://github.com/dzsquared/25-days-of-serverless-day15

Other Resources

https://westus.dev.cognitive.microsoft.com/docs/services/5cd27ec07268f6c679a3e641/operations/56f91f2e778daf14a499f21b
https://www.npmjs.com/package/request-promise-native
https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/quickstarts/node-analyze