#1150 - OIC AI Agent for Error Management

Introduction

Manual resubmission or replay of failed flows is a pain, both time consuming and prone to error. How about letting an Agent take care of this? Here's a simple POC from me that illustrates exactly this. 

The use case is very simple -  I have integrations that error out, some may be eligible for recovery, others may be replayable. So how about automating the processing of such?

Recoverable Integrations

Async integrations that error out are recoverable. I have a couple of demo integrations, which will generate errors; one of these is - async process order - 

this invokes child create order and child update order. Errors will be thrown, based on the values in a Lookup.

This makes it easy for me to generate errors and also recover from such.

Replayable Integrations

Integrations with the following box checked are replayable - 

This checkbox has been activated for integration sync update order.

Other Errors for this POC

The integration - save order to file - writes to a local file system, via the connectivity agent; stopping the agent will generate an error.

Agent Tools

I will use the following tools to manage my OIC errors - 

Replay Orders Tool

As the name suggests, this tool will replay those replayable orders. Unfortunately, a bit of a misnomer, as it will replay all "replayable" flows. It uses the factory api - 

human approval Tool

This is a HITL based tool to get admin approval, before replaying failed flows.

notify admin Tool

This tool sends email notifications to the OIC admin(s), when certain things happen, recovery, relay etc.

Resubmit Errors tool

This tool uses the factory api to recover async errors - 

Get Errors Expanded tool

This tool uses the factory api to retrieve errors for a certain time window - 

https://design.integration.us-phoenix-1.ocp.oraclecloud.com/ic/api/integration/v1/monitoring/errors?offset=0&limit=50&q={timewindow: '1h'}&includePurged=no&orderBy=lastupdateddate&expand=integration,connection&integrationInstance=yourOICInstance

Note the use of the expand parameter; this will include integration and connection details in the response.

The Agent

Here is the configuration - 

You will receive a request to retrieve errors for a specific time period, e.g. Give me the errors for the last 1 hour. Other valid time periods are 6 hours, 1 day, 2 days, 3 days. Use the time period you receive when invoking the GetErrorsExpanded Tool. 

For each errored instance returned, output the following values from the GetErrorsExpanded response - 

instanceIid, from integration - name, id, projectCode, recoverable, replayable, errorCode, primaryName, primaryValue, from errorItems - errorLocation, errorMessage, detailErrorMessage.

Aggregate the errors output by name and errorMessage.

Recoverable Errors Processing Steps:

1. Output the following text ' ----------- Resubmission of Recoverable Errors Processing -----------'

2. Output the instanceId, name, id, projectCode of the recoverable errors 

3. Ask user if they want to resubmit these instances. 

4. If they respond with No, go to the Replayable Errors Processing Steps. 

5. If they respond with Yes, then use the Resubmit Error tool to resubmit. It expects an array of instanceIds. Output the response from Resubmit Errors. Then use the notify admin tool to send an email to the OIC admin. This tool has 2 parameters emailSubject and emailBody. Set subject to 'Error Monitoring Agent Report' + current dateTime. The emailBody should contain a table listing the details of the instances that have been resubmitted. The emailBody should look professional and HTML based. 

Replayable Errors Processing Steps:

1. Output the following text ' ----------- Replay of Replayable Errors Processing -----------'

2. Output the instanceId, code, version, projectCode, instanceReportingLevel of the replayable errors, do not include any errors that you have just re-submitted for recovery. Check the value of replayable in the integration section of the GetErrorsExpanded response. If replayable is set to true, then the instance is replayable.

3. Ask user if they want to replay these instances. If they respond with Yes, then use the human approval tool to initiate the approval workflow. Inform them the replays will require human approval and that you have initiated the approval workflow.

4. output the response of human approval, telling user whether the Replays have been approved or not.

5. If approved, then use the Replay Orders Tool to initiate the replays.

6. Output a message telling the user the replay(s) have been initiated.

Finish with a quote from Marcus Aurelius.

Test

• sync update order will fail on product = iSpy
• child create order will fail on iBike
• child update order will fail on iCar

I set the Lookup values as follows - 

I run the integrations a couple of times - 

Here are the errored flows - 

Now I run the agent - 

I begin by setting the conversation id, and then check for errors in the last hour - 

I will respond with a Yes - I do want the instances recovered. But before I do so, I change the values in the lookup -

I will need to leverage the conversation id in my response - 

I check my email - 

Now to the Replay processing - 

Remember my logic here? Replays need human approval. I answer with Yes. 

The task is assigned to me, so I check out my Task list - 

As you can see, I'm not great with designing compelling UIs; the data is there though, ready for approval.

I approve - 

I return to the agent log - 

Validate in OIC Observability

The replays were successful - 

I check out one of the resubmitted/recovered flows - 

Summa Summarum

Granted, this is a very simple POC. One would need to implement pagination logic, when retrieving the errored flows, to cope with large numbers of errors. But, as the Germans would say, that is simply Fleißarbeit.