Use Case
We need to upload big documents with a file size of up to 2GB from SFTP to Salesforce via MuleSoft. In Salesforce, the target object is ContentVersion. Normally, to upload a document to Salesforce, we can use the Create or Update operation from the Salesforce connector provided in MuleSoft. However, for such a big document, the Create or Update operation won’t be able to process and will throw below error:
Couldn’t execute action over partner client because a connection exception has occurred. Runtime to handle the connection exception. Root exception was: com.sforce.ws.SoapFaultException: Maximum size of request reached. Maximum size of request is 52428800 bytes.
This limitation has been mentioned in detail here MuleSoft Help Center.
As also suggested by the above link, to upload documents up to 2GB to Salesforce, we need to use Salesforce Rest API to “Inserting a ContentVersion” – Salesforce Developers, which can be done by using HTTP Request connector.
However, here comes our main problem. By using HTTP Requester to call Salesforce Rest API to upload a big document, you will soon face the below error because processing a big payload requires a lot of memory.
java.lang.OutOfMemoryError: Java heap space
Upscaling the app by increasing vCore size may allow you to suppress the issue for a short term but the longer the app runs the higher the memory amount it uses and eventually the error will come back.
The best solution is to configure the app to stream the payload of the document via HTTP Request connector, so it will consume the least amount of memory at any time.
How to config Mule app to stream payload
MuleSoft provides different streaming strategies such as File-Stored Repeatable Stream, In-Memory Repeatable Stream, and Non Repeatable Stream… More details on MuleSoft streaming strategies can be found here Streaming Strategies | Mulesoft Documentation .
In our app, both SFTP connector and HTTP connector support the streaming strategies provided by MuleSoft. So which strategies should we use?
In this demo, we assume that we only need to read the payload once, so we can use Non Repeatable Stream to have the best performance.
However, if we consider the error handling and possibly need to re-read the payload, then Repeatable Stream is recommended.
As I tested these strategies, as long as the streaming settings works, Repeatable Stream also has very good performance.
To have our Mule app support streaming, all of the components involved within the processing flow should be configured to support the streaming strategy. These components include SFTP connector, any Transform Message, and HTTP Connector.
Config streaming for SFTP connector in MuleSoft
By default, SFTP connector is already enabled to read Files as a stream. The selected streaming strategy by default is Repeatable file store stream.
However, if you want to change the streaming strategy, it can be done easily by selecting the Streaming Strategy option under the Advanced tab.
Config streaming for Dataweave
If the app doesn’t use any Transform Message component or Dataweave to process the payload, we can ignore this step. However, for our use case, we need to construct the payload request for Salesforce Rest API with Dataweave. Therefore, according to Streaming in Dataweave | MuleSoft Documentation we need at least to have deferred = true settings in our Dataweave code to stream the output of Dataweave.
Below is example of the request payload for Salesforce Rest API to create Content Version.
%dw 2.0
import * from dw::core::Binaries
output multipart/form-data deferred=true, boundary = "boundary_string"
---
{
parts: {
entity_content: {
"Content-Disposition": {
"name": "entity_content",
},
headers: {
"Content-Type": "application/json"
},
content: {
"Title": "dummy-file",
"PathOnClient": "dummy-file" ++ vars.fileExtension,
}
},
VersionData: {
headers: {
"Content-Disposition": {
"name": "VersionData",
"filename": vars.fileName
},
"Content-Type": "application/octet-stream"
},
content: payload
}
}
}Config streaming for HTTP connector in MuleSoft
Finally, we need to enable Streaming in HTTP Request connector in order to stream the Request payload instead of sending it as a whole.
Within the HTTP Request Connector Configuration, we can look at below settings:
- Request streaming mode (under Settings tab): this defines if we want to stream the Request body or not. By default, the connector will only stream the InputStream. For our use case, we can set it as AUTO (Default) or ALWAYS
- Stream Response (under General): this defines if we want to stream our Response payload or not. By default, it is not enabled. As this will not impact our use case, we can enable it or just leave it as is.
- Streaming strategy (under the Advanced tab in the HTTP Request operation): Similar to the SFTP connector that we mentioned above, this defines the streaming strategy we want to use within the app. By default Repeatable file store stream is selected, but we can select Non repeatable stream for our use case.
In addition to the above settings done within the HTTP Connector, we also need to adjust the Mule engine to not wait for the whole payload in the HTTP Requester. This can only be done by adding the below JVM parameter.
-M-Dmule.http.requestStreaming.enableNote: this JVM Parametter is very important. Without it, the app will not stream data as expected.
In Anypoint Studio, this can be set in the Run Configuration of our project as below
When deploying the app to Cloud Hub, similar settings also need to be set in the Properties tab.
How to test streaming strategies in Anypoint Studio locally
By following the above settings, our app is ready to run and will be able to stream data from SFTP to Salesforce via HTTP Request. The expected result is that when we increase the size of the file, the app will be able to upload to Salesforce without facing Out Of Memeory error.
However, how can we test it and make sure that the app is running correctly and we can see the difference between streaming and no streaming?
Due to the length of the content, we decided to split it into another article where we can show you in detail of steps and demo on how to do it. Please stay tuned for the article update.