iPaaS@ORACLE.CLOUD: August 2023

Kudos to my colleague Valeria C. and the OCI Logging Analytics team for creating these compelling dashboards. These dashboards are based on the OCI Service Metrics relevant for OIC, and the OIC log data pushed to OCI Logging Service.

It's maybe a good idea to check out my previous posts on OCI Logging and OCI Logging Analytics, if you're totally new to this area -

OCI Logging Analytics

OCI Logging

Lig dúinn tosú, as we say in Ireland -

The dashboards are available to you now, just log in to your OCI tenancy ->

Let's kick the tyres -

Oracle Integration: Health Overview

Note the first widget - Environments - this is fleet management in practice.

Environments refer to the individual OIC instances, the logs of which are being forwarded to OCI Logging Analytics. See the final summary section for details of how this happens.

These instances can be OIC Gen2 or OIC3 instances.

So the initial view I see, when opening the dashboard, is over multiple instances. I also have the ability to filter down to individual OIC instances -

Note, the gen2 instance surfaces via its name, in my case, oicpm. The OIC3 instance is identified by its OCID.

Also note, one can filter on individual integrations as well -

You select the time period that's relevant to you -

and one can also activate auto-refresh -

Finally, note the ability to drill into a widget -

This is the Log Explorer view - you see the query behind the widget -

'Log Source' = 'OCI Integration Activity Stream Logs' | stats distinctcount(Instance) as Instances by Identifier | sort -Instances

You can play around with this and, if you're happy the result, save it as a new widget.

The query language is simple enough - check out the OCI Logging Analytics docs here.

Summa summarum - from the general to the particular!

Dashboard Widgets

This dashboard includes the following widgets -

Environments - my OIC instance fleet
Integrations - number of distinct integrations that have been executed in the time period selected e.g. sync_SFDC_contact_to_ERP, createNewCustomer_Netsuite etc.
Integration Instances - Number of instance flows in the selected time period.
Instances in Error Hospital - number of errored instances available for re-submission.
Instances in Error Hospital - bar chart representation of the above.
Instance By Environments - see at a glance the work distribution over multiple OIC instances
Instances By Integration ID - see at a glance the load distribution over integrations e.g. 30% of my flows are for the integration createNewOrder-Netsuite. Business is good!
Instances in Catch Fault - details instances where faults have been caught via OIC Fault Handlers.
Number of Inbound Requests - number of request processed by the OIC engine - this will also include integrations called via local invoke
Inbound Request Processing Time P95 - see at a glance, how integrations are performing.
Number of Outbound Requests - check out the number of invokes made from OIC integrations e.g. the integration createNewCustomer_Netsuite invoking Netsuite.
Outbound Request Invocation Time - check out the performance of your outbound invokes.
Potential Issues - this is a very interesting widget that trawls the logs for potential issues - let's look at some -

New Issues:

Here we are trawling the logs for errors, faults etc.

Outliers:

These are issues/events that occurred only once and may point to anomalies.

Clusters:

Log entries with a similar pattern

The Potential Issues widget is extremely useful for both reactive and proactive error monitoring.

Summary

This is a dashboard provided to you out of the box by OCI Logging Analytics. As you've seen already, many of the dashboard widgets are powered by queries one can view/edit in Log Explorer. Net, net, everything is configurable, from the widget names, to the underlying queries etc. Simply make a copy of the OOTB Dashboard and configure it to suit your specific needs -

Oracle Integration Key Metrics

This dashboard is based on the OCI Service metrics for OIC.

Filtering is available, as in the previous dashboard -

Configuration again is possible, just duplicate the dashboard and edit to suit your specific requirements. For example, integrations with Fusion ERP may be critical for you, so you can create a widget for that specific adapter. You could make this even more specific - the Fusion ERP adapter usage by a particular integration e.g. createOrderFusionERP.

Oracle Integration Time Taken Analysis

Here you see at a glance, how long integrations are taking to execute. This is very useful, as you immediately see the most time consuming integrations, based on the widget bubble size.

From this high level view, I can easily drill down to the individual instance flows -

Summary

Here you have 3 extremely useful dashboards, all you need to do is ensure your OIC logs are getting to OCI Logging Analytics. This is very simple, here we go -

Enable Logging for your OIC Instance - this is done via OIC management in the OCI Console -

Note the Log Name and Log Group - in my case - DEV_OIC_PM3_ActivityStream and OIC_DEV_PM3LogGroup

Now to OCI Logging Service -

here we have a Service Connector that pushes the data to OCI Logging Analytics.

Note the log names and groups -

As you can see, logs from 3 OIC instances are being pushed to the same log group. These 3 instances are on OIC3. The fourth log is for a gen2 instance. All of these logs are being forwarded to OIC Logging Analytics, via the Service Connector.

It's that easy - so get analysing!

Simple use case here - users created in SFDC are pushed to a downstream app. If a user is deleted/deactivated in the app, the SFDC pendant needs to be deactivated.

But this post will cover more features of the adapter, including use of the SFDC Bulk API - however, you have to use the Resource Owner Password Credentials security policy or Authorization Code Credentials in order to use V2 of the SFDC Bulk API.

I take this as an opportunity to check out this security policy -

For ROPC, I will need the SFDC client id / secret.

This is how I get such -

I begin by creating a new Connected App in SFDC -

The callback url has the following format - https://hostname from EndpointURL/icsapis/agent/oauth/callback

Now to the scopes - essentially specifying what the app will be allowed to do in SFDC.

I just give full access, at this stage -

I click Save and then -

Now back to the connection definition in OIC - I use the key and secret.

So why would I use Resource Owner Password Credentials? Again back to us wanting to use v2 of the SFDC Bulk API.

Here is the difference, compared to Salesforce Username Password Policy -

Actions available are the same -

but let's check out the Bulk Data Operations -

here are the options - Bulk v2.0 will be used later on in this post.

Compare this to Salesforce Username Password Policy -

Great that we got that started. The next section details a simple example of updating an SFDC user. Essentially, I retrieve the user using SOQL and update it via the adapter CRUD functionality. Net, net a simple select and update.

Updating User in SFDC

So t the simple user update - this can be done easily using the OIC adapter for SFDC -

In the following sample, I retrieve active users from SFDC - one of these, Julius Martov, will be updated -

As you can see, Julius is active -

Here is the simple integration flow -

GetSFDCUsers is configured with the following SOQL -

SELECT ID, CommunityNickname, EMAIL, FirstName, LastName, IsActive from User

Note the testing capabilities of the adapter wizard -

The Switch action filters out the Mencheviks and sets them to deactivated -

I also set the alias, just for the sake of it.

The Active flag is set as follows -

Let's test it out -

Validate in SFDC -

Julius is now known informally as the mart, and yes, the user has been deactivated.

Now let's do the same using the SFDC Bulk api v2.

Updating User in SFDC using Bulk API

Now to the Bulk API implementation - The ROPC based connection I created earlier, will be used.

Bulk api is essentially asynchronous file upload. My first challenge was getting the correct file format for SFDC Users. In the end, I did an export, selecting only the User object -

Per default, include all data is checked. I de-selected this and just select User -

The exported file is as follows -

I make one change to the file, setting alias to "the martian".

This will now be my input file for bulk load. I copy this to the OIC File Server -

Now to the upload integration -

Here I read the file using the ftp adapter, then invoke the bulk load via the SFDC adapter. The latter is configured as follows -

I map a couple of fields - ID, Alias, IsActive etc.

I then add another Bulk API invoke via the adapter - Final Batch - indicating I have no more files to process. SFDC Bulk API has a size limit on files, so you could end up having multiple input files, hence FinalJob.

I run a test - and check in SFDC -

Looks like we have an issue, so let's create another integration to check out the error.

Simple logic here, we invoke the SFDC bulk api - getFailedRecords - this returns the error(s) and data as a .csv file. This file is stored in the OIC VFS (virtual file system), hence the next action being Stage File - List Files. Then each error file is written to the OIC File Server via the ftp adapter.

The SFDC Invoke is configured as follows -