#1174 - Processing large files in OIC

Introduction

This post is just a couple of notes, primarily for myself, but maybe of interest to others.

The use case is simple - 

• list files in /inbox on OIC File Server
• for each file -
• invoke an async integration to write the file to an on-premise directory

This is implemented by a scheduled integration that uses the OIC File Server Native Action to list the files in /inbox. It then invokes an async integration for each file; it that does the following -

• check file size
• if >= 10MB
• use the FTP adapter to get the file reference from OIC File Server
• else
• use the OIC File server native action to get the file reference from OIC File Server
• file ref used as input to File Adapter - Write File - invoke

Test 1

The async integration is as follows -

Here I use the file server native action to get the file ref. I then map to target as follows - 

I run the test - 

Scheduled integration lists 2 files - 

Only 1 is written to my on-premise directory - 

I check out the activity stream for the largeFile processing - 

I amend the async flow as follows - 

I use the FTP adapter (download) operation to get the reference to the large file. Naturally, the FTP connection is to my OIC File Server.

I run the integration again - 

Note, in the mapper for LargeFileWrite I just map source fileRef to target FileRef, i.e. no encode to ref(decode to ref) logic. This logic is required for the other path though.

Net, net, we need to use the FTP adapter for files greater than 10MB.