#1115 - 26.01 New Features - OIC Agent Request Correlation

Introduction 

This is a new feature in 26.01, the ability to link requests via a correlation id. 

Let's start with a simple order request -

The payload definition comes from the prompt template - 

Note, the variable fields are enclosed in {{variable}}

create order for {{nationality}} customer {{customerName}} for the product, {{productName}}, price is {{productPrice}}. Customer email is {{customerEmail}}

Back to our test - 

Now we create a new request, adding the correlation id from the first one.

#1114 - OCI API Gateway and OIC Monitoring

Introduction

We have many Oracle Integration customers who front OIC with OCI API Gateway. Ergo incoming requests to OIC, initially pass through OCI API Gateway, before being routed to OIC.

Wouldn't it be great to be able to monitor those requests as they traverse both OCI Services? This is possible because of a correlation id which is shared between both services. This id is based on the opc-request-id. The latter is a unique, Oracle-assigned identifier for a specific API request, used for tracing, auditing, and troubleshooting.

This post will detail the mechanics of how to actually implement this monitoring.

But firstly, a big thanks to Sreeji from the OCI Log Analytics team for his support here.

Backend OIC Integration

In this scenario, the OIC Integration invoked is called processOrders. It calls another integration, validateOrder, via local invoke.

API Gateway Setup

Here is my API Gateway Deployment setup -

Ensure logging is enabled - 

I execute the request to the gateway -

Check out the logs in OCI Logging

Now, in OCI Logging, I see the api gateway and oic logs. Note the first log entry shown is from api gateway, routing the request to OIC. Note the last entry is from api gateway, processing the OIC response.

 

The api gateway log has the following format - 

Note the opcRequestId entry - 

"/14A3FC5552D04597A8EDF81EECFB635D/7D3278EF2D014AC4B359D70ADCA03A09"

Now to the OIC activity stream entry - 

Note the opcRequestId entry - 

"5TYRCGIW24W2CNSPKVG7JC5FAE84P1XU/14A3FC5552D04597A8EDF81EECFB635D/O6TIS1HTURTM4M3HMR232FLC5W9DYWOA"

As you can see, the first part of the api  gateway opcRequestId surfaces as the second part of the OIC opcRequestId. 

We need to create Correlation IDs that reflect this relationship in OCI Log Analytics.

OCI Log Analytics

Getting API Gateway logs into Log Analytics

I create a new Log Group for api gateway execution log in OCI Log Analytics -

I then create a Connector in OCI Logging to push the api gateway log data to OCI Log Analytics - 

Creating the Correlation ID field for the API Gateway log

The example field entry -

/14A3FC5552D04597A8EDF81EECFB635D/7D3278EF2D014AC4B359D70ADCA03A09

The regex - \/{Correlation ID:.*}\/

Creating the Correlation ID field for the OIC Activity Stream Log

The example field entry -

5TYRCGIW24W2CNSPKVG7JC5FAE84P1XU/14A3FC5552D04597A8EDF81EECFB635D/O6TIS1HTURTM4M3HMR232FLC5W9DYWOA

The regex - .+/{Correlation ID:.+}/

Now to creating the Log Explorer Query.

Log Analytics - Log Explorer Query

Query - 'Correlation ID' != null | link includenulls = true 'Log Source', 'Correlation ID', 'OPC Request ID', Identifier | stats unique(Instance) as 'Instance ID', avg(Duration) as 'Avg. Duration' | eval 'Avg. Duration' = unit('Avg. Duration', second) | rename 'Group Duration' as 'Time Taken' | sort 'Correlation ID', 'Start Time' | fields -'OPC Request ID'

Let's look at these columns in detail - 

The Log Explorer query can be saved and added to a dashboard. I did this - 

Note, I don't see the tabular data. For this I need to punchout to Log Explorer - 

The times shown can be easily verified in OIC Observability, let's execute another request from Postman and check out the dashboard.

I zoom in on the times - 

I check out the activity stream for the first instance id in OIC Observability - 

Message Received at 08:57:34.108

Reply sent at 08:57:34:680

Duration = 572 msecs.

The second instance id - 

Duration = 119 msecs.

We see the duration for OCI API Gateway logs is 705 msecs. There are 2 log messages here, one when the request is initially processed and routed to OIC, and the second, logging the response from OIC.

The duration here is 705msecs, which gives us an CI API Gateway overhead from - 

705 - (572 + 119) = 14 msecs.

Summa Summarum

This simple demo shows the power of OCI Log Analytics for monitoring requests as they traverse different OCI services. I hope this provides you with the wherewithal to create related dashboards in this space.

 

#1113 - Extracting Billing Costs per Integration from OIC

Introduction

Here is a simple example, from which you can extrapolate - the basis is the following Project and it's integrations - 

validateOrder is only invoked via local invoke by the integrations - processOrderSync and processOrderAsync.

ProcessLargeOrderListSync - processes a file of 80KB.

Hourly Orders Run is a scheduled job that does essentially nothing, except WAIT for 10 seconds.

Let's run these integrations - 

Now to OCI Dashboards - 

As you can see, I have 2 widgets - OIC Integration Technical Message Count by Integration and Billable Messages by Integration

The former shows the number of "technical" messages / requests OIC received, while the latter shows the resulting billable messages. Both widgets are grouped by integration.

So why are there differences between the two? Let's investigate - 

The scheduled integration - Hourly_Orders_Run executed twice. However, all this flow does is WAIT for 10 seconds. It does not contain any invokes that retrieve data. If it did, only data > 50KB is counted, from a billing perspective. Ergo, no billable messages for this integration.

The integration, ProcessLargeOrdersListSync, has run twice. However, the input file of both occasions was 82KB, this results in 2 billable messages per flow.

ProcessOrder ran 5 times and generated 5 billable messages. 

ProcessOrderAsync ran 3 times and generated 3 billable messages. 

The integration, ValidateOrder, ran 8 times, but all of these were via local invoke from ProcessOrder and ProcessOrderAsync. Ergo, these are not billed.

Behind each widget lies a query, based on OCI service metrics.

For the "Technical" messages widget - 

MessagesReceivedCount[1h]

{resourceId=$(params.oic-env)}.grouping(FlowCode).sum()

For the "Billable" messages widget - 

BilledMessageCount[1h]

{resourceId=$(params.oic-env)}.

grouping(IntegrationFlowIdentifier).sum()

This Dashboard was created in OCI Log Analytics and includes the following filters - 

So how did I add the filters? I simply duplicated one of the OOTB Integration dashboards, delivered with OCI Log Analytics -

The OOTB dashboards are those created by Oracle - 

Open the Health Overview dashboard - 

Click on the Actions dropdown and select Duplicate - 

Give it a name -

Open the Dashboard - 

Click Edit and delete all the existing widgets in the dashboard -

All gone! - 

Now create a new query based widget - Action - Edit - 

Copy and paste the "technical" query - 

Note the $(params.oic-env)} - this is the link to the OIC resource id in the filter. We need to add this parameter to this widget - 

The default Visualisation is Table, let's change this - 

Amend as follows - 

This results in -

Do the same for the "Billing" widget.

A very nice feature is the ability to export to CSV - 

Policies Required

add your OCI user to a group, e.g. OIC-OCI-LogAnalyticsGroup.

add the following policies to allow group access to OCI Log Analytics etc.

allow group OIC-OCI-LogAnalyticsGroup to read metrics in compartment yourCompartment

allow group OIC-OCI-LogAnalyticsGroup to manage log-content in compartment yourCompartment

allow group OIC-OCI-LogAnalyticsGroup to manage management-dashboard-family in compartment yourCompartment

allow group OIC-OCI-LogAnalyticsGroup to manage loganalytics-features-family in tenancy

allow group OIC-OCI-LogAnalyticsGroup to manage loganalytics-resources-family in compartment yourCompartment

allow group OIC-OCI-LogAnalyticsGroup to read compartments in compartment yourCompartment

allow group OIC-OCI-LogAnalyticsGroup to MANAGE alarms in compartment yourCompartment

allow group OIC-OCI-LogAnalyticsGroup to MANAGE ons-topic in compartment yourCompartment

Summa Summarum

You can augment the dashboard with other billing related metrics -