#954 OIC and OCI Logging Analytics

One of the compelling advantages of OIC, apart from being the best EiPaaS ever, is it's membership of the OCI family. Today I am looking at OCI Logging Analytics and the value add it brings to OIC flow monitoring and log analysis. OIC itself does provide extensive monitoring capabilities and this is complemented by the analytical capabilities of OCI Logging Analytics, which I will often refer to as OCI LA. 

Firstly, a big thank you to my colleague, Valeria C. for her ideas and support here!

In respect of OIC, OCI LA has 2 main sources data souces -

• OCI Service Metrics
• OIC Activity Stream logs.

This post will cover how you can create rich dashboards for your OIC instances in OCI LA.

Before we begin - a quick review of OIC Logging levels - this is important as this determines what gets sent to OCI LA.

OIC Logging Levels

Production - All activities outside loops and invoke/logger activity inside loops (up to a 1000 iterations) are shown in the activity stream and this data is retained for a period of 32 days. 

Audit - Additionally wire payloads of trigger/invoke(s) are shown in the activity stream and this data is retained for a period of 8 days.

Debug - Additionally all payloads and all activities inside loops (up to a 1000 iterations) are shown in the activity stream and this data is retained for a period of 24 hours. It may slowdown your system and also pose a security risk as sensitive information from payload will be logged.

Here's a simple example of Audit output in OIC - this simple integration creates a customer in Netsuite.

Note, I see the payload passed to Netsuite - 

Now I activate the integration with Production level logging, I then run the same test - As you can see, only the activities executed are logged - no payloads etc.

Let's switch to OCI Logging and see what data we're getting there. Note, you can enable your OIC instance to auto-push the logs to OCI Logging.

OIC Logs in OCI logging

Here's the log for 1 of the createCustomer Netsuite flows, with trace set to production -

 

Let's check out the message structure - the salient content is in red. 

{

  "datetime": 1677082743000,

  "logContent": {

    "data": {

      "actionName": "CreateCustomerService",

      "actionType": "Receive",

      "executedTime": "2023-02-22T16:18:48.074Z",

      "instanceId": "lh_Dt7LMEe2W88lIN_tLLQ",

      "integrationFlowIdentifier": "AA_CREATENETSUITECU!01.00.0000",

      "message": "Message started processing",

      "userId": "NIALL..."

    },

    "id": "9629d681-b2cc-11ed-98ed-77be3985b31d",

    "oracle": {

      "compartmentid": "ocid1....",

      "ingestedtime": "2023-02-22T16:19:04.020Z",

      "loggroupid": "ocid1...",

      "logid": "ocid1.log.oc1.phx.amaaaaaajfhkgfyasvzbmk24tiizm5ybpcyoygwto4f6zaa5rtwfqdyuhbcq",

      "tenantid": "ocid1.tenancy.oc1..aaaaaaaagg3lwhfrb3klicoqffeacxqg6akvwadohdjlnoh262dnz7a3ak5q"

    },

    "source": "ocid1.integrationinstance.oc1.phx.amaaaaaajfhkgfya3ew3fmwmm2fzfqmcgre5pj6uqon2sdqolgbbqymbhdtq",

    "specversion": "1.0",

    "time": "2023-02-22T16:19:03.000Z",

    "type": "com.oraclecloud.integration.integrationinstance.activitystream"

  }

}

Let's look at the trace set to Audit and run -

Note, the payload is now available in the Message field - 

This is important, as this is also the log format in OCI Logging Analytics (OCI LA). OCI LA allows us to parse the logs so we could extract payload data from this field and use it in dashboards, e.g. top products ordered etc. 

You create a service connector in OCI Logging to push the OIC logs to OCI LA.

OCI Logging Analytics - Components

Dashboards - this is the main page for viewing/editing Dashboards -

Log Explorer - used for creating queries on top of logs e.g. the OIC logs. These queries can then be saved and visualised in dashboards.

Administration - manage the meta data, driving the dashboards. 

OCI Logging Analytics - Service Metrics Dashboard

OCI Service Metrics include the following - 

Here is a sample dashboard - 

Filters are a really cool feature in LA -

 

Consider we can push logs from multiple OIC instances to LA, filtering allows us to have a fleet overview, as well as the option to drill down into a particular OIC instance and even integration.

I have 2 OIC instances - Dev and Prod - 

I won't go into detail on all the widgets - but here is a quick intro - say I want the execution time metrics for a specific integration - 

I now select a specific integration and the widget updates - 

Now on to a Dashboard based on the OIC logs.

OCI Logging Analytics - OIC Log Dashboard

Let's look at these widgets in more detail -

These 3 widgets are self-explanatory - The rendering is via Tiles. The first tile gives us the count of OIC instances monitored. The second one shows how many flows have executed in the Dashboard timeframe. Note the trend graph here. Here is the actual query behind it - 

  

Here is the actual query - 'Log Source' = 'OCI Integration Activity Stream Logs' | stats distinctcount(Instance) as 'Flow Instances', trend(distinctcount(Instance))

Here is the menu in Log Explorer -

Save/Save as... also gives one the opportunity to save a query visualisation to an existing Dashboard.

Now on to some really cool widgets - 

Let's drill into this - 

This is available OOTB from OCI Logging Analytics - LA does some intelligent parsing of the logs, providing us with a great start in respect of error finding/resolution.

Here is another compelling widget - this provides me with intel on the execution times (duration) of my integrations over a period of time.  

 

For example - my Netsuite createCustomer integration - 

Dashboards can also be interactive - let's check this out with the following widget - 

Clicking on the icon brings me to Log Explorer - 

Note the query - 'Log Source' = 'OCI Integration Activity Stream Logs' | fields Message | fields Identifier, Action, 'Action Type', Instance, -'Log Source', -'Entity Type' 

Now say I want to search the logs for a specific string e.g. 123. I just need to amend the query as follows - 

'123' and 'Log Source' = 'OCI Integration Activity Stream Logs' | fields Message | fields Identifier, Action, 'Action Type', Instance, -'Log Source', -'Entity Type'

Summa Summarum

OCI LA is a must have for OIC Fleet Management. Try it out, today!