Tuesday, October 14, 2025

#1088 - Monitoring ATP, as part of overall OIC Monitoring

This is a follow up to my blog post - giving OIC Monitors access to salient data. You can find the post here.

I was talking with a customer this morning who mentioned they also need to monitor ATP. They have many integrations that invoke PLSQL stored procedures running on their ATP instance.

For them, the first step was to monitor CPU Utilization and to create an alarm on the relevant OCI Service Metric. So how to do such? 

Click the Monitoring tab on your ATP home page in OCI to see all of the ATP Service Metrics.

If you're not the OCI Admin, then your user/group will need access to service metrics, alarms and notifications. The relevant policies etc. are discussed in the post mentioned above. So please go thru that now and ensure you have sufficient permissions.

Back to ATP -

I click on view all database metrics.

I will begin by creating an alarm on CPU Utilization. The alarm will be set to fire on 80%+ utilization. It will result in an email being sent to ATP admin.

The pre-requisite here is a Notification.

Create Subscription - 
I choose email, then add the email addresses of my ATP admins -

Admin(s) confirm subscription - 

Back to the ATP Service Metrics - 

Click Create an alarm on this query
 - 


As you can see, currently my peak is ca. 20%.


Select the topic -

and your done!

Summa Summarum

Many OIC customers make extensive use of ATP as a business DB. Thus they need to monitor what's going on there, as well as what's happening on the OIC front.

Next step would be to create OCI Dashboards and/or OCI Log Analytic widgets for ATP and thus provide one stop dashboards for admins. Please check out my other Observability posts on how to do such.












.

.

.








  

Posted by at No comments:
Labels:

Tuesday, October 7, 2025

#1087 - OIC Scheduled Jobs monitoring

Introduction

Most, if not all, OIC customers use scheduled integrations, as there are many such requirements in the integration space e.g. daily upload of new orders to Fusion ERP etc. FBDI is key to Fusion apps, so scheduled integrations have a large role to play here. This post will look at monitoring scheduled jobs, via the factory api. The reason for doing such may be to get salient data, create mashups from that data or export the data to 3rd party apps for further processing. Typical questions are -
  • what is the mean execution time for certain scheduled integrations?
  • are my scheduled integrations executing on time?
  • are all scheduled runs being executed?
OIC Observability will answer most of these, i.e. within OIC itself you get the answers to many questions. However, you may need to extract key execution data and import it into a 3rd party app, such as Splunk. You could also have other questions that are currently not covered by OIC Observability.

I'll begin with a couple of facts about scheduled integrations and also some best practices. Then we get to our demo project, which contains a couple of scheduled jobs. Finally, on to the factory api to extract salient data.

Facts and Best Practices 

  1. Scheduled Integrations are treated as asynchronous integrations, thus adding to the service limit - concurrent asynchronous requests.
  2. This limit is dependent on the number of message packs you have assigned to your OIC instance.
  3. Schedule integrations execute as singletons, i.e. only one instance of a specific scheduled integration can be active at one time.
  4. Use the following pattern, if non-singleton - scheduled integration --> async integrations(s). In other words, offload the bulk of the work from the scheduled integration to an async integration, if possible. This will ensure your scheduled integration run completes quickly, thus making a more efficient use of resources.
  5. You can define a schedule for your scheduled integrations; try and spread out the runs over the day, i.e. don't have all starting at 17:00 every evening, if possible from a business perspective.
  6. The next run is queued after the previous run completes. This may cause issues, if your schedule is too tight, e.g. run every 10 minutes. For example - Future Runs are -
    • 10:00 am
    • 10:10 am
    • 10:20 am
    • etc.
        The 10 am run takes longer than 10 minutes, this will cause the 10:10 am run to be skipped.

Sample Project and using the Monitoring Factory APIs

 
Here is my sample project - 

The integration processInvoicesBatch should run every 2 hours; processOrdersBatch is scheduled to run every hour. Both include a wait, initially set to 300 seconds - 

I check out the future runs page in project observability - 

Retrieve aggregated data for each scheduled integration


Now let's get some high level data via the factory api - 

https://design.integration.us-phoenix-1.ocp.oraclecloud.com/ic/api/integration/v1/monitoring/integrations?offset=0&limit=999&q=%7BprojectCode%3A+%27AA_PROJECT2%27%2Ctimewindow%3A%271d%27%2C+status%3A%27ACTIVATED%27%7D&orderBy=time&return=monitoringui&integrationInstance=yourOICInstance

Here is the response - 

"dataFetchTime": "2025-10-07T08:36:22.204+0000",
    "items": [
        {
            "code": "PROCESSORDERSBATCH",
            "flowStatus": "ACTIVATED",
            "id": "PROCESSORDERSBATCH|01.00.0000",
            "lastUpdated": "2025-10-07T08:32:53.195+0000",
            "lastUpdatedString": "7 Oct 2025 08:32:53 GMT",
            "links"...
            "mepType": "MEP07",
            "name": "processOrdersBatch",
            "nextRunDateString": "No Data",
            "noOfAborted": 0,
            "noOfErrors": 0,
            "noOfMsgsProcessed": 0,
            "noOfMsgsReceived": 1,
            "noOfSuccess": 0,
            "optimizedVersion": "1.0",
            "projectCode": "AA_PROJECT2",
            "scheduleApplicable": true,
            "scheduleDefined": true,
            "scheduleStatus": "ICS_SCHEDULE_ACTIVE",
            "successRate": 0,
            "version": "01.00.0000"
        }
    ],
    "links"...
    "timeWindow": "24",
    "totalResults": 1
}

The first run is scheduled for 10:42:52 - this timestamp doesn't appear in the initial response.

I wait until the first run starts and execute the api call again - 

"dataFetchTime": "2025-10-07T08:43:34.020+0000",
    "items": [
        {
            "code": "PROCESSORDERSBATCH",
            "flowStatus": "ACTIVATED",
            "id": "PROCESSORDERSBATCH|01.00.0000",
            "lastUpdated": "2025-10-07T08:42:52.476+0000",
            "lastUpdatedString": "7 Oct 2025 08:42:52 GMT",
            "links":...
            "mepType": "MEP07",
            "name": "processOrdersBatch",
            "nextRunDateString": "No Data",
            "noOfAborted": 0,
            "noOfErrors": 0,
            "noOfMsgsProcessed": 0,
            "noOfMsgsReceived": 1,
            "noOfSuccess": 0,
            "optimizedVersion": "1.0",
            "projectCode": "AA_PROJECT2",
            "scheduleApplicable": true,
            "scheduleDefined": true,
            "scheduleStatus": "ICS_SCHEDULE_ACTIVE",
            "successRate": 0,
            "version": "01.00.0000"
        }
    ],
    "links":...
    "timeWindow": "24",
    "totalResults": 1
}

Now the first run has completed and the second run is queued - 

back to the api - 

{
    "dataFetchTime": "2025-10-07T08:52:05.217+0000",
    "items": [
        {
            "code": "PROCESSORDERSBATCH",
            "flowStatus": "ACTIVATED",
            "id": "PROCESSORDERSBATCH|01.00.0000",
            "lastUpdated": "2025-10-07T08:47:52.732+0000",
            "lastUpdatedString": "7 Oct 2025 08:47:52 GMT",
            "links":...
            "mepType": "MEP07",
            "name": "processOrdersBatch",
            "nextRunDateString": "No Data",
            "noOfAborted": 0,
            "noOfErrors": 0,
            "noOfMsgsProcessed": 1,
            "noOfMsgsReceived": 2,
            "noOfSuccess": 1,
            "optimizedVersion": "1.0",
            "projectCode": "AA_PROJECT2",
            "scheduleApplicable": true,
            "scheduleDefined": true,
            "scheduleStatus": "ICS_SCHEDULE_ACTIVE",
            "successRate": 100,
            "version": "01.00.0000"
        }
    ],
    "links":...
    "timeWindow": "24",
    "totalResults": 1
}

the lastUpdated timestamp is the completion timestamp. I see 1 message successfully processed and 2 messages received.

Now the second run has completed and the 3rd is queued - 

{
    "dataFetchTime": "2025-10-07T09:03:09.877+0000",
    "items": [
        {
            "code": "PROCESSORDERSBATCH",
            "flowStatus": "ACTIVATED",
            "id": "PROCESSORDERSBATCH|01.00.0000",
            "lastUpdated": "2025-10-07T09:02:52.654+0000",
            "lastUpdatedString": "7 Oct 2025 09:02:52 GMT",
            "links":...
            "mepType": "MEP07",
            "name": "processOrdersBatch",
            "nextRunDateString": "No Data",
            "noOfAborted": 0,
            "noOfErrors": 0,
            "noOfMsgsProcessed": 2,
            "noOfMsgsReceived": 3,
            "noOfSuccess": 2,
            "optimizedVersion": "1.0",
            "projectCode": "AA_PROJECT2",
            "scheduleApplicable": true,
            "scheduleDefined": true,
            "scheduleStatus": "ICS_SCHEDULE_ACTIVE",
            "successRate": 100,
            "version": "01.00.0000"
        }
    ],
    "links": ...
    "timeWindow": "24",
    "totalResults": 1
}

You get the idea. 

Retrieve instance data for each scheduled integration flow


Now to another api - retrieving each instance

https://design.integration.us-phoenix-1.ocp.oraclecloud.com/ic/api/integration/v1/monitoring/instances?offset=0&limit=50&q=%7Btimewindow%3A%271h%27%2C+includePurged%3A%27no%27%2CprojectCode%3A%27AA_PROJECT2%27%7D&orderBy=lastupdateddate&fields=detail&integrationInstance=yourOICInstance

Here is the current status from project observability -

The api response is as follows - I get 4 results - here is the data for a completed flow - 

{
    "dataFetchTime": "2025-10-07T09:11:40.990+0000",
    "hasMore": false,
    "id": "instances",
    "items": [
        {
            "creationDate": "2025-10-07T08:57:53.453+0000",
            "date": "2025-10-07T09:07:52.363+0000",
            "duration": 299788,
            "flowType": "SCHEDULED",
            "hasRecoverableFaults": false,
            "id": "tbZNIaNbEfCM5rOh3FVr_w",
            "instanceId": "tbZNIaNbEfCM5rOh3FVr_w",
            "instanceReportingLevel": "Production",
            "integration": "PROCESSORDERSBATCH|01.00.0000|AA_PROJECT2",
            "integrationId": "PROCESSORDERSBATCH",
            "integrationName": "processOrdersBatch",
            "integrationVersion": "01.00.0000",
            "invokedBy": "niall.commiskey@oracle.com",
            "isDataAccurate": true,
            "isPurged": false,
       
            "mepType": "SCHEDULED",
            "nonScheduleAsync": false,
            "opcRequestId": "oci-B3DDF17AC79D1BA-202510070832/F8099A9604244C7B99F259BF18E138A6/EAC9569AD2204496B18B4E0248CFA0DC",
            "outboundQueueNames": [],
            "processingEndDate": "2025-10-07T09:07:52.363+0000",
            "projectCode": "AA_PROJECT2",
            "projectFound": true,
            "projectName": "AA-Project2",
            "receivedDate": "2025-10-07T09:02:52.575+0000",
            "replayable": false,
            "replayed": false,
            "status": "COMPLETED",
            "trackings": [
                {
                    "name": "param_wait",
                    "primary": true,
                    "value": "300"
                }
            ]
        },

Here is the data for a scheduled flow -  
        {
            "creationDate": "2025-10-07T09:07:52.340+0000",
            "date": "2025-10-07T09:07:52.340+0000",
            "fifo": false,
            "flowType": "SCHEDULED",
            "hasRecoverableFaults": false,
            "id": "Gq0NFKNdEfCM5rOh3FVr_w",
            "instanceId": "Gq0NFKNdEfCM5rOh3FVr_w",
            "instanceReportingLevel": "Production",
            "integration": "PROCESSORDERSBATCH|01.00.0000|AA_PROJECT2",
            "integrationId": "PROCESSORDERSBATCH",
            "integrationName": "processOrdersBatch",
            "integrationVersion": "01.00.0000",
            "invokedBy": "niall.commiskey@oracle.com",
            "isDataAccurate": true,
            "isLitmusFlow": false,
            "isLitmusSupported": false,
            "isPurged": false,
            "lastTrackedTime": "2025-10-07T09:07:52.340+0000",
            "links": ...
            "litmusResultStatus": "",
            "mepType": "SCHEDULED",
            "nonScheduleAsync": false,
            "opcRequestId": "oci-B3DDF17AC79D1BA-202510070832/B2065CFB86E94F549B34CD5489FC5B32/3569DF4A925A44D486AE282C1EC2F70C",
            "outboundQueueNames": [],
            "projectCode": "AA_PROJECT2",
            "projectFound": true,
            "projectName": "AA-Project2",
            "replayable": false,
            "replayed": false,
            "scheduledTime": "2025-10-07T09:12:52.000+0000",
            "status": "SCHEDULE_WAITING",
            "trackings": [
                {
                    "name": "Primary",
                    "primary": true,
                    "value": "undefined"
                }
            ]
        },
scheduledTime is the time this should start running, I check in the activity stream -   

It started executing at 11:12:52:493

The 2 hour difference is because of UTC vs CET.

The scheduledTime value is only returned when the status is SCHEDULE_WAITING.

So let's just recap, before we continue -The first api gives us an aggregated overview of the schedule integration, the second, one record per instance.  

Retrieve the future runs schedule

The next api is used to retrieve the future runs schedule - 

https://design.integration.us-phoenix-1.ocp.oraclecloud.com/ic/api/integration/v1/monitoring/futureruns?q=%7Bstartdate%3A%272025-10-05+22%3A00%3A00%27%2Cenddate%3A%272025-10-12+21%3A59%3A59%27%2CprojectCode%3A%27AA_PROJECT2%27%7D&returnAll=true&timezone=Europe%2FBerlin&integrationInstance=yourOICInstance

The result - 
{
    "items": [
        {
            "code": "PROCESSORDERSBATCH",
            "futureRuns": [
                {
                    "runTime": "2025-10-07T11:42:52.971+0200"
                },
                {
                    "runTime": "2025-10-07T11:52:52.971+0200"
                },
                {
                    "runTime": "2025-10-07T12:02:52.971+0200"
                },
                {
                    "runTime": "2025-10-07T12:12:52.971+0200"
                },
                {
                    "runTime": "2025-10-07T12:22:52.971+0200"
                },
                {
                    "runTime": "2025-10-07T12:32:52.971+0200"
                },
                {
                    "runTime": "2025-10-07T12:42:52.971+0200"
               


We can now bring all these together - Let's invoke the 3 apis from an integration -

I begin with the Aggregated stats - that the following api - 
https://design.integration.us-phoenix-1.ocp.oraclecloud.com/ic/api/integration/v1/monitoring/integrations?offset=0&limit=999&q=%7BprojectCode%3A+%27AA_PROJECT2%27%2Ctimewindow%3A%271d%27%2C+status%3A%27ACTIVATED%27%7D&orderBy=time&integrationInstance=yourOICInstance

My integration switches on integration type, as I'm only interested in scheduled jobs -

Next step is to get salient data on executed runs - 

I log the following data - 

Integration: PROCESSORDERSBATCH Instance Id: E2wGsKNrEfCM5rOh3FVr_w StartTime: 2025-10-07T10:52:52.317+0000 EndTime: 2025-10-07T10:57:52.642+0000 Duration (msecs): 300325 Status: COMPLETED

Duration in msecs is 300325 which translates to 5 minutes 325 msecs.


Here is the activity stream view for this instance - 

Next step is getting the timestamps for the future runs - 



Summa Summarum

Granted this is a very basic implementation, however, it does give us the salient information we require for monitoring scheduled integrations.

The high level data tells us how many instances have executed and what the outcomes were ('COMPLETED' etc.)

The next invoke gives us intel on what has already executed - when the run started, how long it took, and whether it was successful or not.

The final invoke gives us intel on the future runs schedule. With all this data, we could work out min/max/mean execution times. We can also see whether the schedule is being adhered to, i.e. whether runs are executing on time.












Posted by at No comments:
Labels:

Friday, October 3, 2025

#1086 - OIC Observability - View Slowest Activities

Introduction

This feature has been available for some time now; it's availability doesn't jump out in your face, so let's look at it in some detail.

View Slowest Activities 

This feature is available in instance tracking. The goal here is to give you insight into where most of the flow execution time is being spent in respect of the actions you have defined in your integration.

Just to make sure there is no ambiguity, we can look at the execution time as being divided into 2 areas -

  1. The time taken to execute the actions you have defined within your integration. In the following case that would be - Map(writeFile), File server(writeFile) and Map(saveFile). The trigger is an exception here. Let's classify this as infrastructure. 
  2. infrastructure - triggers, transitions between actions etc.
As you see above, the integration has been run with the trace level set to debug. Let's check out the slowest activities.

Here we see that writing the file to OIC File Server, via the file server native action, consumed almost half the execution time.

I run the same integration again, this time with trace set to audit - 

As you can see, the Map actions are not included. 

Production level trace is the same - 

Summa Summarum

This feature is extremely useful, especially in the development phase, where you are testing in debug mode. It is also very useful in production, for seeing where processing time is being spent, in respect of invokes. You can always flip between production / debug tracing to get more data - 

If you need data over a longer time period, then look at OCI Service Metrics, e.g. Outbound Request Invocation Time. Here you can investigate the time taken by your invokes over whatever period of time you want. You also get the rich variety of stats -


 
 


 







 

   

Posted by at No comments:
Labels:
Subscribe to: Posts (Atom)