#1183 - OIC Service Metrics and where to monitor them

Introduction 

I've blogged extensively about monitoring dashboards, but there's always either some new metrics or new aspects that I haven't covered in depth; hence today's post.

I'll look in depth at the service metrics, including those related to billing. I will also discuss the options one has in respect of dashboarding.

The post covers OCI Console Dashboards, OCI Management Dashboards and OCI Log Analytics. My goal is to provide insight and guidance, so you know which tool to use for your specific monitoring needs.

The starting point is the OIC instance page in the OCI console; here you see all the OIC metrics, currently 19 of them -

The default time filter applied is the last hour; this is important to note, as metrics can be gathered at different times.

For example, I just ran a load test and see the following -

However, at this point in time, the Total consumed messages metric shows 0 -

I have to wait until the next metric harvest, which is at 07:00 am - 

This number tallies with the run count from SOAP UI - 

Net, net - the takeaway is some metrics get updated quicker than others. Billing metrics, for example, are always harvested on an hourly basis.

I have the following options for each metric -  

Create an alarm on this query - OCI alarms can be easily configured. In this case, I may want an alarm if the Total consumed messages exceeds 10k. Check out my post on such for more details.

I can also view the underlying query in Metric Explorer - 

Clicking advanced mode gives me direct access to the MQL.

This is useful, when I want to create more function rich dashboards in other OCI services. 

To conclude, the monitoring page, for your OIC instance in OCI, gives us 19 key metrics for OIC tech and billing health. 

It offers multiple time interval filters - 

and the ability to easily create alarms. However, the widget rendering is basic and, summa summarum, this is not the tool for LOB users; more a great ad hoc tool for OIC admins.

With the basics behind us, let's look at what OCI Console Dashboards offer us.

OCI Console Dashboards 

 

Think of this as the first level of dashboarding available to you. I have already created a dashboard; let's add a new widget to it - 

I click configure to select the metric, set filters etc.

Very simple - I'll add a couple more - 

Note how some metrics have more dimensions than others. In this case, I can also monitor the messages received for a specific integration. 

I can also choose other chart types, such as bar chart. 

Back to queries underlying the widgets, let's look at the following - 

Here I see messages received by multiple OIC instances in my compartment - 

I can restrict this to a specific OIC instance as follows - 

I will filter on the resourceid (OCID) of a specific OIC instance - 

The OCID can be found in the About Box in your OIC design-time - 

Note the difference between the 2 widgets above. Successful Messages is filtered on resource id, while async queue depth is not, thus one sees the 3 OIC instances listed. 

I now apply the resourceid filter to this widget as well - 

Back at Dashboard level, note the actions available - 

I can copy the url and make it available to other stakeholders. You will need to create a policy with the following statements -

Allow group OICDashboardViewers to read dashboards-family in compartment DashboardCompartment

Allow group OICDashboardViewers to read metrics in compartment OICCompartment where target.metrics.namespace='oci_integration'

To summarize, OCI Console Dashboards provide richer widgets than what we get with the OOTB service metrics. There is also no charge for using these dashboards and they can provide the basis for monitoring your OIC instances. However, the ability to drill down from an OIC fleet view to individual OIC instances, needs to be hard-coded, via resourceid dimension settings; but if this is not a primary need, then these dashboards could suffice. 

OCI Management Dashboards

These dashboards can be used for Performance analysis and diagnosis across OCI observability services. 

The permissions required are as follows - 

Allow group DashboardViewers to read management-dashboard-family in compartment DashboardCompartment

Allow group DashboardViewers to read metrics in compartment OICCompartment where target.metrics.namespace='oci_integration'

The above is one of the widgets, I created in a custom billing dashboard. But, before we get into that, these dashboards can be used for fleet monitoring. Let's check out the filters - 

The filters selected here can be used in the underlying widget MQLs. I got these filters by duplicating one of the OOTB OIC dashboards - 

Each of these contains the filters, so I simply duplicate one of these OOTB dashboards and use that as the basis for my custom dashboard.

Management dashboards can also surface OIC activity stream data, but this post is dedicated to service metrics, so I will not be covering activity stream here. Check out my plethora of posts on OCI Log Analytics for such.

Back to my Billing Metrics dashboard - the first widget is the following - 

This is the fleet view, I see my 3 OIC instances - dev, test and prod.

Note we only see 2 lines, that's because two of the OIC instances have the same number of message packs.

I only need to click on dev instance, for example, to see it's data - 

Let's check out widget creation - 

we begin by editing the dashboard - 

What's a widget group? A widget group in OCI Management Dashboards is a named container that holds related widgets within a dashboard. It helps organize the layout—for example, keeping all OIC performance charts together. You can move, resize, duplicate, or ungroup the container.

Let's create one for OIC Fleet level Billing Metrics - 

I drop the existing widget into the newly created group -

Now I add a new widget based on the Total consumed message packs metric - the MQL I coy form the OIC service metrics page - 

BilledMessagePackCount[60m].sum() 

Now to the MQL - 

Click Run - 

The data is shown in tabular format, per default, but we can change the visualisation - 

The result - 

Hover over to see the exact values - 

Let's add another widget group for billing metrics at the OIC Instance level - 

We have billing metrics for - 

• integrations
• decisions
• RPA
• HITL and Process
• extended data retention
• ORCL Mgd DR

But let's begin with the instance level metrics - 

The MQL - ConfiguredMessages[60m]{resourceId = $(params.selectedInstance)}.grouping().sum()

Note the $(params.selectedInstance). This is a variable I create in the widget and link to the dashboard filter - 

This greatly eases drill downs to individual OIC instances.

As you hopefully can see, I've added 3 widgets to the OIC instance level billing metrics group - 

• Configured messages 
• Message throughput
• Total consumed message packs

I now add a further widget group - Top integrations for selected OIC instance. 

Note the MQL I will use - 

BilledMessageCount[1h]{resourceId = $(params.selectedInstance)}.groupBy(integrationFlowIdentifier).sum().top_n(10)

The second widget is just a variation, with top_n(5).

Finally I add a widget group - Billed message by component for selected OIC instance. 

I fill it with the following -

To summarize OCI Management Dashboards - it's free, easy to use, allows for much richer MQL queries. It also supports many more visualisations compared to OCI Console Dashboards. Finally, it supports the fleet to instance drill down OOTB.

OCI Log Analytics

The dashboard from the previous section has a great name - Oracle Integration: Working NiallcBlogs Billing Dashboard_copy

Let's now go to OCI Log Analytics - 

As its name suggests, this service is for analyzing logs such as the OIC activity stream.

But let's check out the Dashboards -

The dashboards from OCI Management Dashboards are also surfaced here.

I can, of course, create dashboards here in OCI Log Analytics, with the same modus operandi as OCI Management Dashboards.

The Log Explorer component allows us to drill into logs, such as the OIC activity stream -

The default query can be easily replaced, with the following, for example - 

'Log Source' = 'OCI Integration Activity Stream Logs' and Duration != null | eval DurationMs = unit(Duration, ms) | stats count as Samples, avg(DurationMs) as AvgMs, pct(DurationMs, 95) as P95Ms, max(DurationMs) as MaxMs by Integration, Action | sort -P95Ms | head 15

This will give us some latency intel on the integrations.

I can change the visualizations - 

I can save this as a widget - 

I've also added it to the dashboard I created previously.

Net, net - with OCI Log Analytics I can include service metric and OIC activity stream based widgets in the same dashboard.

This is a huge value add. 

OCI Log Analytics pricing is storage-based, so how much you retain and for how long drives the cost. 

Summa Summarum

People often ask me - which are the important service metrics for OIC? Essentially, to paraphrase George Orwell, they are all important, but some are more important than others. What matters most to me is how many messages are flowing through and how efficiently they’re being handled. From a metrics perspective, we could initially look at - 

• MessagesReceivedCount
• MessagesSuccessfulCount
• MessagesFailedCount
• AsyncInboundRequestsDepth

To this we could add a custom MQL for concurrent sync request processing - 

NumberOfInboundRequests[1m]{resourceId = “yourOIC OCID”}.groupBy(serviceInstanceOcid)

.sum()/60 * InboundRequestProcessingTime[1m]{resourceId = “yourOIC OCID"}.

groupBy(serviceInstanceOcid).mean()/1000

Note, InboundRequestProcessingTime in respect of async requests, measures how long OIC takes to persist the incoming request and send an acknowledgment. It reflects the trigger’s handling of the request, not the full integration execution time.

Also note, OIC has service limits for async and sync concurrent requests. This is message pack based - 1 MP = 100 concurrent sync / 50 concurrent async. 2 MPs - 200 concurrent sync / 100 concurrent async and so on. 

Let's create a simple dashboard for this - 

I now run a load test from SOAP UI. This invokes an async integration, that, in turn, invokes a sync integration.

The sync integration includes a WAIT action set to 60 secs; thus the async queue will build up.

I start a test with 4125 runs - 

I check in OIC Observability - 

Here we see the async concurrency limit in action; only 51 sync flows  have being started.

I check the dashboard -

The second tranche of sync flows has been started.

The async queue is building up.

Concurrent sync processing is ca. 50.

We see the concurrent sync metric leveling off at ca. 50. 

We know that our concurrent sync request processing limit is 100, as we have 1 message pack assigned to the service instance. We also see we have a large async queue. Here we can improve throughput processing by increasing the number of message packs assigned to the instance. On the other hand, maybe I'm ok with the throttling, I just want insight into what's happening. 

Regarding the actual execution times for the async integration, you can check the stats in OIC project observability - 

Back to my dashboard, I could also add a widget for the Outbound Request Invocation Time - this will reflect the call’s round-trip time while the parent waits for the sync integration to respond.

Here I select P95 as the stat and I see the result is a constant 60k msecs, which accurately reflects the 60 second WAIT in the sync integration.

I can amend the MQL to make this specific to my async integration - OutboundRequestInvocationTime[5m]{resourceId = $(params.selectedInstance), integrationFlowIdentifier = "ASYNC_CREATE_ORDER!01.00.0000"}.grouping().percentile(.95)

To conclude, we have a plethora of OIC service metrics monitoring capabilities in OCI. Net, net - you decide what to use, based on your specific needs.

Happy Monitoring!