#882 Installing OIC Connectivity Agent on OCI Compute

 

This is a follow up from the previous OCI Streaming post - the OCI Streaming connection requires a connectivity agent to be specified, when one uses the connection as a trigger. I tried this out with the connectivity agent installed on my laptop. Naturally, this is not feasible for any organization, so installation of the agent on an OCI Compute is a compelling alternative.

Step 1 - Spin up an OCI Compute Instance - 

 

 

 

You can accept the defaults - 

Regarding keys -  simplest way to go is Generate a key pair for me. Save the public / private key locally.

then Paste the public key and click Create.

Step 2 - Login to the Compute Instance - 

I am using Putty to login to the compute instance; pre-requisite is the private key .ppk format. So I use PuttyGen to load the key generated by OCI and then save it in the Putty format.

 Now I leverage this in the Putty connection -

Step 3 - Install JDK on the Compute Instance - 

sudo yum install java

Step 4 - Create a Directory for the Connectivity Agent -

 

As you can see, I have copied the agent installer zip to this directory, using WinSCP.

Step 5 - Create a new Agent Group in OIC - 

then configure the InstallerProfile.cfg file to point to this Agent Group.

Step 6 - Leverage in a Connection Definition - 

#881 - OIC Pub/Sub with OCI Streaming Part 3

This final post in the series covers using the OCI Streaming adapter as a Trigger. 

This requires us to specify a connectivity agent in the connection definition. My Streaming api endpoint is public, so the connectivity agent can run anywhere e.g. on my laptop.

 

Here I leverage my agent group - NIALLC_AGENTGROUP. Now to the integration with OCI Streaming as trigger - 

I apply the same processing logic as in the previous posts - just a LOG action - 

I activate and test - create a new product in Shopify - 

The integration that subscribes to the Shopify New Product event fires and publishes the message to the Stream.

 

The Streams Subscriber integration is triggered.

#880 OIC Pub/Sub with OCI Streaming Part 2

 Leading on from the previous post...

Here we have multiple subscribers, let's call them cg01 and cg02 - cg standing for consumer group. The previous post had 1 subscriber, now we are looking at 2. This is very easy to implement. One just needs separate integrations per subscriber, it's what you would require anyway. So let's check out my OIC artefacts -

AA-StreamingProducts-Subscribe is configured to subscribe as consumer cg01.

AA-StreamingProducts-Subscribe-CG02 is configured to subscribe as consumer cg02.

Just to recap - the "publish" configuration is as follows - 

Now for a couple of tests to prove this works as we expect. I add a new product in Shopify - 

I see the subscription to the new Shopify has fired and the publish integration has done it work - 

I run the cg01 "subscribe" integration - 

I now run the cgo2 "subscribe" integration - same result - 

So now I create 2 new products - Keg of Tucherbraeu and Keg of Doom Bar. I then run the "subscriber" for cg01. 

Ok, looks good. Now I add another product - Keg of Harp. 

I run the cg01 "subscribe" integration - 

it continues from the appropriate offset - 

I now run the cg02 "subscribe" integration - 

As you an see, I get the 3 new products added since the last "cg02" run.

Conclusion

It is very easy to do Pub/Sub with OIC and OCI Streaming. Regarding consumers - I need one integration per consumer. I do not need to create the consumers first in OCI Streaming. I just need to specify the consumer group when configuring the OCI Streaming invoke. OCI Streaming will check if the consumer group exists, if so, it will continue from the relevant offset, otherwise it will create the new consumer group. There is currently a limit of 50 consumers at OCI Streams level, this can be increased, if required. 

Finally, this example mentions various beers, please don't treat this as a recommendation to try out any of them, although I can vouch for their thirst-quenching powers.     

#879 - OIC Pub/Sub with OCI Streaming

Simple use case here - publish and subscribes to events from Shopify. 

Normally I would be doing this with orders, however, my trial Shopify account doesn't allow me to create orders, so I'm leveraging Products here. But, I know you are all experts at extrapolating.

Now someone may think of using Pub/Sub pattern available when creating a new integration in OIC. However, as you can see - it has been deprecated.

 

The publish to OIC integration style has been deprecated. Oracle recommends that you use app driven orchestration and use an invoke to publish a message into the OIC streaming service.

So let's take that advice -

Here is my Stream - ProductStream-NiallC

Now to the Connection creation in OIC - Please see my Shopify blog posts for details on how to create that connection. So now to the OCI Streaming connection creation - 

 Bootstrap Servers entry can be found here - 

As you can see, it also contains the SASL username. You need to generate an Auth Token for the SASL Password - 

Check out the OIC OCI Streaming adapter docs for the steps to create the Trust Store. Test and Save the Connection - 

Now to the "Publish" integration - very simple really - 

I get the JSON for the PublishProduct request payload from the Shopify api docs here

Configuration of the PublishProduct action is as follows - 

I activate and test by creating a new product in Shopify and then check out the stream - 

So that's the Publish part addressed. Now to the subscribe - for this I use a Scheduled Integration - 

The above is the final result - let's look at the integration in detail - 

Subscribe2NewProduct is configured as follows - 

Now to the message structure - 

 

This I copy from the Recent Messages view from Streaming - 

The only logic I add is a LOG action - to output the product name - 

Now to testing the Subscribe - 

I add 2 new products in Shopify -

I run the scheduled integration -