CipherInputStream to the rescue: Saving memory usage when encrypting huge binary payloads

Hi, There:

If you are one of those lucky ones in the northeast like me, I hope your arms are not so sore after shoveling the snow! Normally I enjoy using my jumbo snow blower for any snow that’s larger than 4 inches. But it decided to quit with busted sheer bolts this time! Before it is fixed, my arm and back will step in for the job!

Anyway, at work we need to encrypt huge binary payload due to the nature of the data. I found that simply using Cipher to encrypt large payload, it often busts the Heap Size when the payload is a couple of hundreds of MB, depend on the JVM and the encrypting parameters, of course. But this limit is not desirable for us where we need sometimes 1 GB of data to be encrypted, in a timely way.

At this point, JDK 8 already has Streaming Encryption implemented for us! How nice!  The class is CipherInputStream. Here is the demo code excerpt that will encrypt byte[] mydata through the Streaming process and save the cipher into a file. The time to encrypt is almost linear to the size of byte[] mydata, indicating a nice Streaming process. Most importantly, no more Memory issues with heap size busting!

byte[] salt = new byte[8];
// random SALT

new Random().nextBytes(salt); 

// random Initial Vector IV when cipher init
PBEParameterSpec pbeParamSpec = new PBEParameterSpec(salt, 1000); 

PBEKeySpec pbeKeySpec = new PBEKeySpec(“funkypassword”.toCharArray());
SecretKeyFactory keyFac = SecretKeyFactory.getInstance(“PBEWithHmacSHA256AndAES_256”);
SecretKey pbeKey = keyFac.generateSecret(pbeKeySpec);

// Create PBE Cipher
Cipher pbeCipher = Cipher.getInstance(“PBEWithHmacSHA256AndAES_256”);
// Initialize PBE Cipher with key and parameters
pbeCipher.init(Cipher.ENCRYPT_MODE, pbeKey, pbeParamSpec);

AlgorithmParameters algParams = pbeCipher.getParameters();
byte[] encodedAlgParams = algParams.getEncoded();

byte[] mydata = “make believe large payload secrete data here”.getBytes();
FileOutputStream fos = null;
CipherInputStream cis = null;

fos = new FileOutputStream(new File(“C:/temp/secrete.cipher”));
fos.write(encodedAlgParams);
cis = new CipherInputStream(new ByteArrayInputStream(mydata), pbeCipher);

byte[] b = new byte[1024];
int i = cis.read(b);
while (i != -1) {
fos.write(b, 0, i);
i = cis.read(b);
}

cis.close();
fos.close();

 

I am not posing the codes here. But if you want to decrypt huge binary payload, just use the CipherOutputStream in a similar way. It works just as nicely as CipherInputStream.

Cheers, and if we can’t help it, let it snow – let it snow !!

-Tony

Stream Virus Scanner

Hi, There:

Not much too exciting going on for a while. However, today I did some virus scanning process using Symantec Virus Scanner Engine API. The goal is to streaming huge data binary into Symantec Engine for scanning without busting memory.  It is not fancy but thing working pretty well for a few hundred MB of binary, with decent performance. Due to the streaming, the size limit will be the configured memory upper limit of the Scan Engine itself.

I am outlining the steps here with some example code. Hope it helps.

  • Prepare Scan Engine with Policy (default shown)
ScanEngine.ScanEngineInfo scanEngTobeUsed = new ScanEngine.ScanEngineInfo("virusscanhost",1344);
Vector <ScanEngine.ScanEngineInfo> scanEnginesForScanning = new Vector <ScanEngine.ScanEngineInfo>();
scanEnginesForScanning.add(scanEngTobeUsed); 
ScanEngine scanEngine = ScanEngine.createScanEngine(scanEnginesForScanning);
ByteArrayOutputStream bao = new ByteArrayOutputStream();
StreamScanRequest scanRequest = scanEngine.createStreamScanRequest("", null, bao, Policy.DEFAULT);
  • Streaming binary data into buffer while sending to the Scan Engine
byte[] b = new byte[4096];
while ( // more binary streaming in ... ) {
   b = ....; // get more binary here into buffer
   scanRequest.send(b);
}
  • Finish and examine the Scan Result
Result result = scanRequest.finish();
// read or examine result here

That’s all and cheers!

-Tony

 

SOAP Webservice Streaming – Memory and CPU usage pattern

Hi, There:

I have recently implemented SOAP Streaming with CXF 3.14 and datahandler. The streaming works really well with huge payload in MTOM. Without detailing on the code in this post, let me show the memory/CPU usage in the application on the server side. The pattern on the client side is very similar too.

See the memory usage when streaming for 1 GB (first small oscillation) and 8 GB (second big oscillation) of payload. The heap memory is constantly using and releasing throughout the streaming process.  Meanwhile the CPU usage is heavy also due to the constant heavy IO, since streaming utilizes heavy temp file storage.

Not showing here, but for non-streaming SOAP MTOM, memory usage is pretty much a big lump without oscillation and its CPU usage is minimum due to limited processing and IO. During non-streaming, peak memory usage is much much higher than that of streaming since the whole payload is processed and contained in memory.

Cheers!

-TY

CPU_Mem_Usage

 

MTOM disappeared when using SOAPHandler ??!!

Hi, There:

I am working on Streaming Attachment with SOAP webservices these days, because I need to send GBs of data in one SOAP invocation. Naturally MTOM/XOP is required so base64Binary can become MIME attachment. Everything works well with both client and server. Streaming GB of binary via the SOAP call was a breeze and memory usage on both Client and Server are kept in minimum. The problem arose when I had to add WSSE with UserNameToken/Password into the SOAP header. The way you would add WSSE header is via Handler. The handler itself works fine and user/password are in the WSSE header and taken nicely by the server. The problem is the XOP/MTOM that had been working just disappeared!  The base64Binary went back into SOAP as an inline data inside the XML as base64 strings. As a result, the Streaming functionality is now out of the question. A couple of hundred MB of payload will yield out of memory exception!

The reason of this happening is basically this. The Handler that adds WSSE header is doing its job AFTER the MTOM/XOP is being processed. When the Handler is processing the SOAP message, it will de-XOP it and push binary payload back to the SOAP before adding the WSSE header. As a result, MTOM is gone and if the payload is too big, out of memory exception happens.

Googling shows there are many others having the same problem. In particular, To shreds explained similar thing and offers a work around solution. The work around basically manually implements the XOP process and pushes binary from SOAP inline content back to XOP attachment. I actually tried to implement that work around. It works but it still has a problem. The processing of move the binary back to XOP requires to load the complete payload content in memory first. That requires again a lot of memory. It is true that the transporting from client to server is back to MTOM. But it kind of defeat the purpose for me because I intended to stream GBs of data without loading whole payload into memory.

Finally, my attention diverted from JAX_WS default implementation to CXF. CXF’s factory model allows the client to add WSSE header first before invoking service. Hence, XOP/MTOM will be intact since that is the last step before leaving for the server’s endpoint.

Here is my working code:

JaxWsProxyFactoryBean factory = new JaxWsProxyFactoryBean();

//MTOM
Map<String,Object> props = new HashMap<String, Object>();
props.put(“mtom-enabled”, Boolean.TRUE);
factory.setProperties(props);

// ensure SOAP 1.2
SoapBindingConfiguration configuration = new SoapBindingConfiguration();
configuration.setVersion(Soap12.getInstance());
factory.setBindingConfig(configuration);

// WSSE
Map <String, Object> outProps = new HashMap<String, Object>();
outProps.put(WSHandlerConstants.ACTION, WSHandlerConstants.USERNAME_TOKEN);
outProps.put(WSHandlerConstants.USER, ClientPasswordCallback.testUserID);
outProps.put(WSHandlerConstants.PASSWORD_TYPE, WSConstants.PW_TEXT);
outProps.put(WSHandlerConstants.PW_CALLBACK_CLASS, ClientPasswordCallback.class.getName());
outProps.put(WSHandlerConstants.MUST_UNDERSTAND, “0”); // false/0 or true/1
WSS4JOutInterceptor wssOut = new WSS4JOutInterceptor(outProps);
wssOut.setAllowMTOM(Boolean.TRUE);
factory.getOutInterceptors().add(wssOut);

// Monitors both direction
factory.getOutInterceptors().add(new LoggingOutInterceptor());
factory.getInInterceptors().add(new LoggingInInterceptor());

// Custom Service Class XYZ
factory.setServiceClass(XYZ.class);
factory.setAddress(“http://localhost:7001/ws/endpoint/location&#8221;);
XYZ client = (XYZ) factory.create();

MyFileUploader f = new MyFileUploader();

DataHandler dh = new DataHandler(new FileDataSource(“c:\10GB.zip”));
f.setDataHandler(dh);
XYZResponse nr = client.streamingUpload(f);

With the above code, you will have both WSSE UserNameToken and Streaming MTOM in SOAP calls. The Streaming works exceptionally well with almost a flat liner in memory usage.

I haven’t tried other WSSE functions such as signing/encryption. Hope I can get to them later this week if needed.

Be good and write good code!

-Tony