Tuesday, October 7, 2025

#1087 - OIC Scheduled Jobs monitoring

Introduction

Monitoring scheduled jobs via the factory api.

Here is my sample project - 

The integration processInvoicesBatch should run every 2 hours; processOrdersBatch is scheduled to run every hour. Both include a wait, initially set to 300 seconds - 

I check out the future runs page in project observability - 

Retrieve aggregated data for each scheduled integration


Now let's get some high level data via the factory api - 

https://design.integration.us-phoenix-1.ocp.oraclecloud.com/ic/api/integration/v1/monitoring/integrations?offset=0&limit=999&q=%7BprojectCode%3A+%27AA_PROJECT2%27%2Ctimewindow%3A%271d%27%2C+status%3A%27ACTIVATED%27%7D&orderBy=time&return=monitoringui&integrationInstance=yourOICInstance

Here is the response - 

"dataFetchTime": "2025-10-07T08:36:22.204+0000",
    "items": [
        {
            "code": "PROCESSORDERSBATCH",
            "flowStatus": "ACTIVATED",
            "id": "PROCESSORDERSBATCH|01.00.0000",
            "lastUpdated": "2025-10-07T08:32:53.195+0000",
            "lastUpdatedString": "7 Oct 2025 08:32:53 GMT",
            "links"...
            "mepType": "MEP07",
            "name": "processOrdersBatch",
            "nextRunDateString": "No Data",
            "noOfAborted": 0,
            "noOfErrors": 0,
            "noOfMsgsProcessed": 0,
            "noOfMsgsReceived": 1,
            "noOfSuccess": 0,
            "optimizedVersion": "1.0",
            "projectCode": "AA_PROJECT2",
            "scheduleApplicable": true,
            "scheduleDefined": true,
            "scheduleStatus": "ICS_SCHEDULE_ACTIVE",
            "successRate": 0,
            "version": "01.00.0000"
        }
    ],
    "links"...
    "timeWindow": "24",
    "totalResults": 1
}

The first run is scheduled for 10:42:52 - this timestamp doesn't appear in the initial response.

I wait until the first run starts and execute the api call again - 

"dataFetchTime": "2025-10-07T08:43:34.020+0000",
    "items": [
        {
            "code": "PROCESSORDERSBATCH",
            "flowStatus": "ACTIVATED",
            "id": "PROCESSORDERSBATCH|01.00.0000",
            "lastUpdated": "2025-10-07T08:42:52.476+0000",
            "lastUpdatedString": "7 Oct 2025 08:42:52 GMT",
            "links":...
            "mepType": "MEP07",
            "name": "processOrdersBatch",
            "nextRunDateString": "No Data",
            "noOfAborted": 0,
            "noOfErrors": 0,
            "noOfMsgsProcessed": 0,
            "noOfMsgsReceived": 1,
            "noOfSuccess": 0,
            "optimizedVersion": "1.0",
            "projectCode": "AA_PROJECT2",
            "scheduleApplicable": true,
            "scheduleDefined": true,
            "scheduleStatus": "ICS_SCHEDULE_ACTIVE",
            "successRate": 0,
            "version": "01.00.0000"
        }
    ],
    "links":...
    "timeWindow": "24",
    "totalResults": 1
}

Now the first run has completed and the second run is queued - 

back to the api - 

{
    "dataFetchTime": "2025-10-07T08:52:05.217+0000",
    "items": [
        {
            "code": "PROCESSORDERSBATCH",
            "flowStatus": "ACTIVATED",
            "id": "PROCESSORDERSBATCH|01.00.0000",
            "lastUpdated": "2025-10-07T08:47:52.732+0000",
            "lastUpdatedString": "7 Oct 2025 08:47:52 GMT",
            "links":...
            "mepType": "MEP07",
            "name": "processOrdersBatch",
            "nextRunDateString": "No Data",
            "noOfAborted": 0,
            "noOfErrors": 0,
            "noOfMsgsProcessed": 1,
            "noOfMsgsReceived": 2,
            "noOfSuccess": 1,
            "optimizedVersion": "1.0",
            "projectCode": "AA_PROJECT2",
            "scheduleApplicable": true,
            "scheduleDefined": true,
            "scheduleStatus": "ICS_SCHEDULE_ACTIVE",
            "successRate": 100,
            "version": "01.00.0000"
        }
    ],
    "links":...
    "timeWindow": "24",
    "totalResults": 1
}

the lastUpdated timestamp is the completion timestamp. I see 1 message successfully processed and 2 messages received.

Now the second run has completed and the 3rd is queued - 

{
    "dataFetchTime": "2025-10-07T09:03:09.877+0000",
    "items": [
        {
            "code": "PROCESSORDERSBATCH",
            "flowStatus": "ACTIVATED",
            "id": "PROCESSORDERSBATCH|01.00.0000",
            "lastUpdated": "2025-10-07T09:02:52.654+0000",
            "lastUpdatedString": "7 Oct 2025 09:02:52 GMT",
            "links":...
            "mepType": "MEP07",
            "name": "processOrdersBatch",
            "nextRunDateString": "No Data",
            "noOfAborted": 0,
            "noOfErrors": 0,
            "noOfMsgsProcessed": 2,
            "noOfMsgsReceived": 3,
            "noOfSuccess": 2,
            "optimizedVersion": "1.0",
            "projectCode": "AA_PROJECT2",
            "scheduleApplicable": true,
            "scheduleDefined": true,
            "scheduleStatus": "ICS_SCHEDULE_ACTIVE",
            "successRate": 100,
            "version": "01.00.0000"
        }
    ],
    "links": ...
    "timeWindow": "24",
    "totalResults": 1
}

You get the idea. 

Retrieve instance data for each scheduled integration flow


Now to another api - retrieving each instance

https://design.integration.us-phoenix-1.ocp.oraclecloud.com/ic/api/integration/v1/monitoring/instances?offset=0&limit=50&q=%7Btimewindow%3A%271h%27%2C+includePurged%3A%27no%27%2CprojectCode%3A%27AA_PROJECT2%27%7D&orderBy=lastupdateddate&fields=detail&integrationInstance=yourOICInstance

Here is the current status from project observability -

The api response is as follows - I get 4 results - here is the data for a completed flow - 

{
    "dataFetchTime": "2025-10-07T09:11:40.990+0000",
    "hasMore": false,
    "id": "instances",
    "items": [
        {
            "creationDate": "2025-10-07T08:57:53.453+0000",
            "date": "2025-10-07T09:07:52.363+0000",
            "duration": 299788,
            "flowType": "SCHEDULED",
            "hasRecoverableFaults": false,
            "id": "tbZNIaNbEfCM5rOh3FVr_w",
            "instanceId": "tbZNIaNbEfCM5rOh3FVr_w",
            "instanceReportingLevel": "Production",
            "integration": "PROCESSORDERSBATCH|01.00.0000|AA_PROJECT2",
            "integrationId": "PROCESSORDERSBATCH",
            "integrationName": "processOrdersBatch",
            "integrationVersion": "01.00.0000",
            "invokedBy": "niall.commiskey@oracle.com",
            "isDataAccurate": true,
            "isPurged": false,
       
            "mepType": "SCHEDULED",
            "nonScheduleAsync": false,
            "opcRequestId": "oci-B3DDF17AC79D1BA-202510070832/F8099A9604244C7B99F259BF18E138A6/EAC9569AD2204496B18B4E0248CFA0DC",
            "outboundQueueNames": [],
            "processingEndDate": "2025-10-07T09:07:52.363+0000",
            "projectCode": "AA_PROJECT2",
            "projectFound": true,
            "projectName": "AA-Project2",
            "receivedDate": "2025-10-07T09:02:52.575+0000",
            "replayable": false,
            "replayed": false,
            "status": "COMPLETED",
            "trackings": [
                {
                    "name": "param_wait",
                    "primary": true,
                    "value": "300"
                }
            ]
        },

Here is the data for a scheduled flow -  
        {
            "creationDate": "2025-10-07T09:07:52.340+0000",
            "date": "2025-10-07T09:07:52.340+0000",
            "fifo": false,
            "flowType": "SCHEDULED",
            "hasRecoverableFaults": false,
            "id": "Gq0NFKNdEfCM5rOh3FVr_w",
            "instanceId": "Gq0NFKNdEfCM5rOh3FVr_w",
            "instanceReportingLevel": "Production",
            "integration": "PROCESSORDERSBATCH|01.00.0000|AA_PROJECT2",
            "integrationId": "PROCESSORDERSBATCH",
            "integrationName": "processOrdersBatch",
            "integrationVersion": "01.00.0000",
            "invokedBy": "niall.commiskey@oracle.com",
            "isDataAccurate": true,
            "isLitmusFlow": false,
            "isLitmusSupported": false,
            "isPurged": false,
            "lastTrackedTime": "2025-10-07T09:07:52.340+0000",
            "links": ...
            "litmusResultStatus": "",
            "mepType": "SCHEDULED",
            "nonScheduleAsync": false,
            "opcRequestId": "oci-B3DDF17AC79D1BA-202510070832/B2065CFB86E94F549B34CD5489FC5B32/3569DF4A925A44D486AE282C1EC2F70C",
            "outboundQueueNames": [],
            "projectCode": "AA_PROJECT2",
            "projectFound": true,
            "projectName": "AA-Project2",
            "replayable": false,
            "replayed": false,
            "scheduledTime": "2025-10-07T09:12:52.000+0000",
            "status": "SCHEDULE_WAITING",
            "trackings": [
                {
                    "name": "Primary",
                    "primary": true,
                    "value": "undefined"
                }
            ]
        },
scheduledTime is the time this should start running, I check in the activity stream -   

It started executing at 11:12:52:493

The 2 hour difference is because of UTC vs CET.

The scheduledTime value is only returned when the status is SCHEDULE_WAITING.

So let's just recap, before we continue -The first api gives us an aggregated overview of the schedule integration, the second, one record per instance.  

Retrieve the future runs schedule

The next api is used to retrieve the future runs schedule - 

https://design.integration.us-phoenix-1.ocp.oraclecloud.com/ic/api/integration/v1/monitoring/futureruns?q=%7Bstartdate%3A%272025-10-05+22%3A00%3A00%27%2Cenddate%3A%272025-10-12+21%3A59%3A59%27%2CprojectCode%3A%27AA_PROJECT2%27%7D&returnAll=true&timezone=Europe%2FBerlin&integrationInstance=yourOICInstance

The result - 
{
    "items": [
        {
            "code": "PROCESSORDERSBATCH",
            "futureRuns": [
                {
                    "runTime": "2025-10-07T11:42:52.971+0200"
                },
                {
                    "runTime": "2025-10-07T11:52:52.971+0200"
                },
                {
                    "runTime": "2025-10-07T12:02:52.971+0200"
                },
                {
                    "runTime": "2025-10-07T12:12:52.971+0200"
                },
                {
                    "runTime": "2025-10-07T12:22:52.971+0200"
                },
                {
                    "runTime": "2025-10-07T12:32:52.971+0200"
                },
                {
                    "runTime": "2025-10-07T12:42:52.971+0200"
               


We can now bring all these together - Let's invoke the 3 apis from an integration -

I begin with the Aggregated stats - that the following api - 
https://design.integration.us-phoenix-1.ocp.oraclecloud.com/ic/api/integration/v1/monitoring/integrations?offset=0&limit=999&q=%7BprojectCode%3A+%27AA_PROJECT2%27%2Ctimewindow%3A%271d%27%2C+status%3A%27ACTIVATED%27%7D&orderBy=time&integrationInstance=yourOICInstance

My integration switches on integration type, as I'm only interested in scheduled jobs -

Next step is to get salient data on executed runs - 

I log the following data - 

Integration: PROCESSORDERSBATCH Instance Id: E2wGsKNrEfCM5rOh3FVr_w StartTime: 2025-10-07T10:52:52.317+0000 EndTime: 2025-10-07T10:57:52.642+0000 Duration (msecs): 300325 Status: COMPLETED

Duration in msecs is 300325 which translates to 5 minutes 325 msecs.


Here is the activity stream view for this instance - 

Next step is getting the timestamps for the future runs - 



Summa Summarum

Granted this is a very basic implementation, however, it does give us the salient information we require for monitoring scheduled integrations.

The high level data tells us how many instances have executed and what the outcomes were ('COMPLETED' etc.)

The next invoke gives us intel on what has already executed - when the run started, how long it took, and whether it was successful or not.

The final invoke gives us intel on the future runs schedule. With all this data, we could work out min/max/mean execution times. We can also see whether the schedule is being adhered to, i.e. whether runs are executing on time.












No comments: