CosmosDb with DocumentDB API through Azure API Management

Question

I am attempting to query my new CosmosDB collection through API management. Once proved out, this will be a front-end for user access to logged data. For that reason, I have the data partitioned by subscription ID. In Azure Portal for the Logs collection of my WebApi DB I see partition key as /api_subscription_key. I have the data going in from API Mgt. -> Event Hub -> Stream Analytics -> Cosmos.

Using the query explorer in Azure portal, I can try a query like:

SELECT * FROM c WHERE c.api_subscription_key = '573a1c65bceb52192c140131'

this brings back expected documents I have been successfully writing to CosmosDB for many days

    [
      {
        "eventenqueuedutctimesecond": "2017-07-27T15:09:02Z",
        "business_unit_key": null,
        "user_key": null,
        "api_message_id": "1718ea66-d225-45ec-b3fc-5daff4c7f426",
        "api_identifier": "21926e9d-9206-42b0-b4b1-7e7f1eb4e7dd",
        "api_id": "58d94cc622be39392343d4b6",
        "api_operation_id": "58e682bde055cd0ba4215d4b",
        "api_adapter_id": "573a1c64bceb520aac127ee5",
        "api_subscription_id": "573a1c65bceb52192c140131",
        "api_policy_id": "64BC4270-54AC-42DA-835C-E285F35BCA81",
        "basic_username": "",
        "message_version": "10",
        "claim_business_unit_key": null,
        "claim_user_key": null,
    ...
        "lasterrorsource": null,
        "lasterrorreason": null,
        "lasterrorscope": null,
        "lasterrorsection": null,
        "lasterrorpolicyid": null,
        "id": "7/27/2017 3:09:02 PM",
        "_rid": "9Fc0ANW4fwAoAAAAAAAADA==",
        "_self": "dbs/9Fc0AA==/colls/9Fc0ANW4fwA=/docs/9Fc0ANW4fwAoAAAAAAAADA==/",
        "_etag": "\"0700d90c-0000-0000-0000-597a020e0000\"",
        "_attachments": "attachments/",
        "_ts": 1501168140
      }...

My CosmosDB instance is plexconnectcosmos. Thru API management and its policies I am POSTing to

https://plexconnectcosmos.documents.azure.com/dbs/WebApi/colls/Logs/docs

with these headers (many vestigial and hopefully having no effect):

[
{
name: "Postman-Token",
value: "756c2c21-ef23-4e5a-a63a-ae6aed961d35"
},
{
name: "Ocp-Apim-Subscription-Key",
value: "a2a05eff128943bc89f62b81a63aa368"
},
{
name: "Accept-Charset",
value: "UTF-8"
},
{
name: "Cache-Control",
value: "no-cache"
},
{
name: "Content-Type",
value: "application/query+json"
},
{
name: "Accept",
value: "application/json;odata=nometadata"
},
{
name: "Accept-Encoding",
value: "gzip,deflate"
},
{
name: "Cookie",
value: "x-ms-gateway-slice=008; stsservicecookie=ests; BIGipServerpmc_rest_webservices_http_prod=1242575370.20480.0000"
},
{
name: "User-Agent",
value: "PostmanRuntime/6.2.5"
},
{
name: "x-ms-date",
value: "Wed, 09 Aug 2017 20:10:09 GMT"
},
{
name: "x-ms-version",
value: "2017-02-22"
},
{
name: "MaxDataServiceVersion",
value: "3.0"
},
{
name: "DataServiceVersion",
value: "1.0;NetFx"
},
{
name: "Api-Message-Id",
value: "12427ae7-7704-44cb-b4af-d7e622898b99"
},
{
name: "Api-Identifier",
value: "461f0c19-8df3-4272-9ac7-c64bb776dd56"
},
{
name: "Api-Id",
value: "58987927bceb5204c4e59168"
},
{
name: "Api-Operation-Id",
value: "598b3c72e055cd14fc3abdd1"
},
{
name: "Api-Adapter-Id",
value: "573a1c64bceb520aac127ee5"
},
{
name: "Api-Subscription-Id",
value: "573a1c65bceb52192c140131"
},
{
name: "Api-Policy-Id",
value: "64BC4270-54AC-42DA-835C-E285F35BCA81"
},
{
name: "X-Basic-Username",
value: ""
},
{
name: "x-ms-documentdb-isquery",
value: "True"
},
{
name: "x-ms-documentdb-query-enablecrosspartition",
value: "False"
},
{
name: "x-ms-max-item-count",
value: "1000"
},
{
name: "x-ms-documentdb-partitionkey",
value: "573a1c65bceb52192c140131"
},
{
name: "x-ms-partition-key",
value: "573a1c65bceb52192c140131"
},
{
name: "Authorization",
value: "type=master&ver=1.0&sig=Ke...Q="
},
{
name: "X-Forwarded-For",
value: "75.39.38.67"
}
]

The response I get back is either

{
    "code": "BadRequest",
    "message": "Partition key 573a1c65bceb52192c140131 is invalid.\r\nActivityId: 61836599-fe4b-4232-b55b-2c568eecc767"
}

or

{
    "code": "Unauthorized",
    "message": "The input authorization token can't serve the request. Please check that the expected payload is built as per the protocol, and check the key being used. Server used the following payload to sign: 'post\ndocs\ndbs/WebApi/colls/Logs\nwed, 09 aug 2017 20:35:41 gmt\n\n'\r\nActivityId: 429....2e2"
}

These seems to give me two problems to solve. In the first place how do I go about troubleshooting this partition? It seems to be from my analysis, it is a valid partition, validating by queries in the portal and the headers "x-ms-documentdb-partitionkey" and "x-ms-partition-key". (I have seen both header names in MS documentation, so I am covering my bases with both.)

The "The input authorization token can't serve the request." message suggests to me something variably wrong in my query. I suspect maybe the data value? My policy is little different from one I use for Azure Table Storage REST API and I never have that problem. I am using my read-only primary key taken from Azure portal and stored in API Management's Named Values:

<policies>
    <inbound>
        <base />
        <set-variable name="Content-Type" value="application/query+json" />
        <set-variable name="x-ms-documentdb-isquery" value="True" />
        <set-variable name="x-ms-documentdb-query-enablecrosspartition" value="False" />
        <set-variable name="x-ms-max-item-count" value="1000" />
        <set-variable name="x-ms-version" value="2017-02-22" />
        <set-header name="Content-Type" exists-action="override">
            <value>@((string)context.Variables["Content-Type"])</value>
        </set-header>
        <set-header name="x-ms-documentdb-isquery" exists-action="override">
            <value>@((string)context.Variables["x-ms-documentdb-isquery"])</value>
        </set-header>
        <set-header name="x-ms-documentdb-query-enablecrosspartition" exists-action="override">
            <value>@((string)context.Variables["x-ms-documentdb-query-enablecrosspartition"])</value>
        </set-header>
        <set-header name="x-ms-max-item-count" exists-action="override">
            <value>@((string)context.Variables["x-ms-max-item-count"])</value>
        </set-header>
        <set-header name="x-ms-version" exists-action="override">
            <value>@((string)context.Variables["x-ms-version"])</value>
        </set-header>
        <!-- MS docs may conflict here. Possibly "x-ms-documentdb-partitionkey" req'd and "x-ms-partition-key" not supported -->
        <set-header name="x-ms-documentdb-partitionkey" exists-action="override">
            <value>@(context.Subscription.Id)</value>
        </set-header>
        <set-header name="x-ms-partition-key" exists-action="override">
            <value>@(context.Subscription.Id)</value>
        </set-header>
        <set-variable name="StringToSign" value="@(string.Format("post\ndocs\ndbs/WebApi/colls/Logs\n{0}\n\n", ((string)context.Variables["x-ms-date"]).ToLowerInvariant()))" />
        <set-variable name="cosmosreadonlykey" value="{{CosmosReadOnlyKey}}" />
        <set-variable name="SharedKey" value="@{
        // https://learn.microsoft.com/en-us/rest/api/documentdb/access-control-on-documentdb-resources#constructkeytoken
        System.Security.Cryptography.HMACSHA256 hasher = new System.Security.Cryptography.HMACSHA256(Convert.FromBase64String((string)context.Variables["cosmosreadonlykey"]));
        return Convert.ToBase64String(hasher.ComputeHash(System.Text.Encoding.UTF8.GetBytes((string)context.Variables["StringToSign"])));
}" />
        <set-variable name="Authorization" value="@(string.Format("type=master&ver=1.0&sig={0}", (string)context.Variables["SharedKey"]))" />
        <set-header name="Authorization" exists-action="override">
            <value>@((string)context.Variables["Authorization"])</value>
        </set-header>
        <set-backend-service base-url="https://plexconnectcosmos.documents.azure.com" />
        <rewrite-uri template="/dbs/WebApi/colls/Logs/docs" />
    </inbound>

Some thins I wonder: Could the returned ActivityId help me get any more details, some how? Even without it, is there some logging in Azure I haven't found that would reveal more details.

If anything is obviously wrong here that I am doing, please someone let me know.

Kai Walter · Accepted Answer · 2017-08-10 06:03:37Z

6

I got it running with some minor adjustments.

<policies>
<inbound>
    <base />
    <set-variable name="Content-Type" value="application/query+json" />
    <set-variable name="x-ms-documentdb-isquery" value="True" />
    <set-variable name="x-ms-documentdb-query-enablecrosspartition" value="False" />
    <set-variable name="x-ms-max-item-count" value="1000" />
    <set-variable name="x-ms-version" value="2017-02-22" />
    <set-variable name="x-ms-date" value="@( DateTime.UtcNow.ToString("R") )" />
    <set-header name="Content-Type" exists-action="override">
        <value>@((string)context.Variables["Content-Type"])</value>
    </set-header>
    <set-header name="x-ms-documentdb-isquery" exists-action="override">
        <value>@((string)context.Variables["x-ms-documentdb-isquery"])</value>
    </set-header>
    <set-header name="x-ms-documentdb-query-enablecrosspartition" exists-action="override">
        <value>@((string)context.Variables["x-ms-documentdb-query-enablecrosspartition"])</value>
    </set-header>
    <set-header name="x-ms-max-item-count" exists-action="override">
        <value>@((string)context.Variables["x-ms-max-item-count"])</value>
    </set-header>
    <set-header name="x-ms-version" exists-action="override">
        <value>@((string)context.Variables["x-ms-version"])</value>
    </set-header>
    <set-header name="x-ms-documentdb-partitionkey" exists-action="override">
        <value>@("[\""+context.Subscription.Id+"\"]")</value>
    </set-header>
    <set-header name="x-ms-date" exists-action="override">
        <value>@( (string)context.Variables["x-ms-date"] )</value>
    </set-header>
    <set-variable name="StringToSign" value="@(string.Format("post\ndocs\ndbs/WebApi/colls/Logs\n{0}\n\n", ((string)context.Variables["x-ms-date"]).ToLowerInvariant()))" />
    <set-variable name="cosmosreadonlykey" value="{{CosmosReadOnlyKey}}" />
    <set-variable name="SharedKey" value="@{
    // https://learn.microsoft.com/en-us/rest/api/documentdb/access-control-on-documentdb-resources#constructkeytoken
    System.Security.Cryptography.HMACSHA256 hasher = new System.Security.Cryptography.HMACSHA256(Convert.FromBase64String((string)context.Variables["cosmosreadonlykey"]));
    return Convert.ToBase64String(hasher.ComputeHash(System.Text.Encoding.UTF8.GetBytes((string)context.Variables["StringToSign"])));
}" />
    <set-variable name="Authorization" value="@(string.Format("type=master&ver=1.0&sig={0}", ((string)context.Variables["SharedKey"]).Replace("&","%26").Replace("+","%2B").Replace("=","%3D")))" />
    <set-header name="Authorization" exists-action="override">
        <value>@((string)context.Variables["Authorization"])</value>
    </set-header>
    <set-backend-service base-url="https://mycosmosdb.documents.azure.com" />
    <rewrite-uri template="/dbs/WebApi/colls/Logs/docs" />
</inbound>
</policies>

the partition key needs to be formatted as an array
date is put into the header and the StringToSign based on the same value
did some hacky URL hex encoding - could be improved with proper hex encoding

edited Aug 10, 2017 at 6:03

answered Aug 10, 2017 at 5:19

Kai Walter

4,1213 gold badges41 silver badges73 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Darrel Miller Over a year ago

This is awesome. Let me see what we can do to make that encoding stuff easier.

Tom Schulte Over a year ago

Thanks, @KaiWalter That partition key formatted as an array is what really got me.

Josh Over a year ago

@DarrelMiller did the APIM team ever look at better encoding options? We have a secret with a plus sign in it. Would love to see native encoding functions in policies

Darrel Miller Over a year ago

@Josh Sadly, I'm not aware of any more work in this area recently.

Collectives™ on Stack Overflow

CosmosDb with DocumentDB API through Azure API Management

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related