SOAP Webservice Streaming – Memory and CPU usage pattern

Hi, There:

I have recently implemented SOAP Streaming with CXF 3.14 and datahandler. The streaming works really well with huge payload in MTOM. Without detailing on the code in this post, let me show the memory/CPU usage in the application on the server side. The pattern on the client side is very similar too.

See the memory usage when streaming for 1 GB (first small oscillation) and 8 GB (second big oscillation) of payload. The heap memory is constantly using and releasing throughout the streaming process.  Meanwhile the CPU usage is heavy also due to the constant heavy IO, since streaming utilizes heavy temp file storage.

Not showing here, but for non-streaming SOAP MTOM, memory usage is pretty much a big lump without oscillation and its CPU usage is minimum due to limited processing and IO. During non-streaming, peak memory usage is much much higher than that of streaming since the whole payload is processed and contained in memory.

Cheers!

-TY

CPU_Mem_Usage

 

Advertisements

SSL Handshake under the hood …

Happy Friday!

We always hear about SSL handshake and routinely use it, but never really want/need to drill down to see what really is going on there. This is a nice handshake sequence description. During my development recently, I used CXF to call SOAP webservices via HTTPS over TLS. I ran into a few “handshake walls” which I eventually resolved. In the process, I found enabling tracer for SSL handshake is very useful. For Java, adding this to the VM params will enable it

-Djavax.net.debug=SSL,handshake

I captured the SSL communication as below. I won’t go through them in details because they are very self descriptive.

Be good and do excellent coding!

===========================================================

adding as trusted cert:
Subject: CN=…….
Issuer: CN=…….
Algorithm: RSA; Serial number: ……
Valid from Tue Nov 07 19:00:00 EST 2006 until Wed Jul 16 19:59:59 EDT 2036
…….
…….

Starting to Invoke Webservice –> https://xxx.xxx/mywebservice Fri Jun 26 09:54:27 EDT 2015
keyStore is :
keyStore type is : jks
keyStore provider is :
init keystore
init keymanager of type SunX509
trustStore is: C:\Program Files\Java\jdk1.8.0_45\jre\lib\security\cacerts
trustStore type is : jks
trustStore provider is :
init truststore
adding as trusted cert:
Subject: CN=Equifax Secure Global eBusiness CA-1, O=Equifax Secure Inc., C=US
Issuer: CN=Equifax Secure Global eBusiness CA-1, O=Equifax Secure Inc., C=US
Algorithm: RSA; Serial number: 0xc3517
Valid from Mon Jun 21 00:00:00 EDT 1999 until Mon Jun 22 00:00:00 EDT 2020

…….
…….
trigger seeding of SecureRandom
done seeding SecureRandom
trigger seeding of SecureRandom
done seeding SecureRandom
Ignoring unavailable cipher suite: TLS_DHE_DSS_WITH_AES_256_GCM_SHA384
Ignoring unavailable cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA
…….
…….

Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
main, setSoTimeout(0) called
Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 for TLSv1
Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 for TLSv1
…….
…….

%% No cached client session
*** ClientHello, TLSv1.2
RandomCookie: GMT: 1418549396 bytes = { 224, 43, 120, 102, 139, 75, 239, 187, 229, 175, 114, 139, 145, 9, 221, 183, 138, 2, 112, 134, 123, 94, 7, 68, 36, 208, 182, 123 }
Session ID: {}
Cipher Suites: [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, ……., TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
Compression Methods: { 0 }
Extension elliptic_curves, curve names: {secp256r1, sect163k1, sect163r2, secp192r1, ……, secp256k1}
Extension ec_point_formats, formats: [uncompressed]
Extension signature_algorithms, signature_algorithms: SHA512withECDSA, SHA512withRSA, ……, SHA1withDSA, MD5withRSA
***
main, WRITE: TLSv1.2 Handshake, length = 207
main, READ: TLSv1.2 Handshake, length = 81
*** ServerHello, TLSv1.2
RandomCookie: GMT: 1418549396 bytes = { 142, 100, 141, 6, 255, 138, 243, 120, 33, 45, 208, 118, 224, 24, 6, 151, 36, 161, 154, 87, 240, 174, 80, 69, 194, 85, 209, 122 }
Session ID: {46, 112, 146, 68, 191, 63, 204, 28, 124, 103, 103, 9, 165, 180, 85, 127, 114, 199, 154, 56, 103, 219, 197, 48, 77, 75, 154, 11, 54, 168, 7, 157}
Cipher Suite: SSL_RSA_WITH_RC4_128_SHA
Compression Method: 0
Extension renegotiation_info, renegotiated_connection: <empty>
***
%% Initialized: [Session-1, SSL_RSA_WITH_RC4_128_SHA]
** SSL_RSA_WITH_RC4_128_SHA
main, READ: TLSv1.2 Handshake, length = 2990
*** Certificate chain
chain [0] = [
[
Version: V3
Subject: CN= …….
…….
…….
]
main, READ: TLSv1.2 Handshake, length = 4
*** ServerHelloDone
*** ClientKeyExchange, RSA PreMasterSecret, TLSv1.2
main, WRITE: TLSv1.2 Handshake, length = 262
SESSION KEYGEN:
PreMaster Secret:
…….
…….
CONNECTION KEYGEN:
Client Nonce:
…….
…….
Server Nonce:
…….
…….
Master Secret:
…….
…….
Client MAC write Secret:
…….
…….
Server MAC write Secret:
…….
…….
Client write key:
…….
…….
Server write key:
…….
…….
main, WRITE: TLSv1.2 Change Cipher Spec, length = 1
*** Finished
verify_data: { 5, 13, 28, 40, 35, 90, 186, 31, 217, 111, 119, 83 }
***
main, WRITE: TLSv1.2 Handshake, length = 36
main, READ: TLSv1.2 Change Cipher Spec, length = 1
main, READ: TLSv1.2 Handshake, length = 36
*** Finished
verify_data: { 48, 164, 101, 77, 74, 77, 125, 12, 151, 111, 77, 111 }
***
%% Cached client session: [Session-1, SSL_RSA_WITH_RC4_128_SHA]
main, WRITE: TLSv1.2 Application Data, length = 450
main, WRITE: TLSv1.2 Application Data, length = 4116
…….
…….
main, READ: TLSv1.2 Application Data, length = 25

Finished Invoking Webservice.1435326869036

===========================================================

MTOM disappeared when using SOAPHandler ??!!

Hi, There:

I am working on Streaming Attachment with SOAP webservices these days, because I need to send GBs of data in one SOAP invocation. Naturally MTOM/XOP is required so base64Binary can become MIME attachment. Everything works well with both client and server. Streaming GB of binary via the SOAP call was a breeze and memory usage on both Client and Server are kept in minimum. The problem arose when I had to add WSSE with UserNameToken/Password into the SOAP header. The way you would add WSSE header is via Handler. The handler itself works fine and user/password are in the WSSE header and taken nicely by the server. The problem is the XOP/MTOM that had been working just disappeared!  The base64Binary went back into SOAP as an inline data inside the XML as base64 strings. As a result, the Streaming functionality is now out of the question. A couple of hundred MB of payload will yield out of memory exception!

The reason of this happening is basically this. The Handler that adds WSSE header is doing its job AFTER the MTOM/XOP is being processed. When the Handler is processing the SOAP message, it will de-XOP it and push binary payload back to the SOAP before adding the WSSE header. As a result, MTOM is gone and if the payload is too big, out of memory exception happens.

Googling shows there are many others having the same problem. In particular, To shreds explained similar thing and offers a work around solution. The work around basically manually implements the XOP process and pushes binary from SOAP inline content back to XOP attachment. I actually tried to implement that work around. It works but it still has a problem. The processing of move the binary back to XOP requires to load the complete payload content in memory first. That requires again a lot of memory. It is true that the transporting from client to server is back to MTOM. But it kind of defeat the purpose for me because I intended to stream GBs of data without loading whole payload into memory.

Finally, my attention diverted from JAX_WS default implementation to CXF. CXF’s factory model allows the client to add WSSE header first before invoking service. Hence, XOP/MTOM will be intact since that is the last step before leaving for the server’s endpoint.

Here is my working code:

JaxWsProxyFactoryBean factory = new JaxWsProxyFactoryBean();

//MTOM
Map<String,Object> props = new HashMap<String, Object>();
props.put(“mtom-enabled”, Boolean.TRUE);
factory.setProperties(props);

// ensure SOAP 1.2
SoapBindingConfiguration configuration = new SoapBindingConfiguration();
configuration.setVersion(Soap12.getInstance());
factory.setBindingConfig(configuration);

// WSSE
Map <String, Object> outProps = new HashMap<String, Object>();
outProps.put(WSHandlerConstants.ACTION, WSHandlerConstants.USERNAME_TOKEN);
outProps.put(WSHandlerConstants.USER, ClientPasswordCallback.testUserID);
outProps.put(WSHandlerConstants.PASSWORD_TYPE, WSConstants.PW_TEXT);
outProps.put(WSHandlerConstants.PW_CALLBACK_CLASS, ClientPasswordCallback.class.getName());
outProps.put(WSHandlerConstants.MUST_UNDERSTAND, “0”); // false/0 or true/1
WSS4JOutInterceptor wssOut = new WSS4JOutInterceptor(outProps);
wssOut.setAllowMTOM(Boolean.TRUE);
factory.getOutInterceptors().add(wssOut);

// Monitors both direction
factory.getOutInterceptors().add(new LoggingOutInterceptor());
factory.getInInterceptors().add(new LoggingInInterceptor());

// Custom Service Class XYZ
factory.setServiceClass(XYZ.class);
factory.setAddress(“http://localhost:7001/ws/endpoint/location&#8221;);
XYZ client = (XYZ) factory.create();

MyFileUploader f = new MyFileUploader();

DataHandler dh = new DataHandler(new FileDataSource(“c:\10GB.zip”));
f.setDataHandler(dh);
XYZResponse nr = client.streamingUpload(f);

With the above code, you will have both WSSE UserNameToken and Streaming MTOM in SOAP calls. The Streaming works exceptionally well with almost a flat liner in memory usage.

I haven’t tried other WSSE functions such as signing/encryption. Hope I can get to them later this week if needed.

Be good and write good code!

-Tony

javax.xml.ws.spi.Provider Choose between org.apache.cxf.jaxws .spi.ProviderImpl and com.sun.xml.internal.ws.spi.ProviderImpl

Hi, long time again!

Spent a few days with this on and off and finally solved the problem. So it is worth posting!  The source of the problem is not universal, probably. But the solution would be useful for many.

For some strange reason, our CXF client fails to talk to a Web server when the payload is more than 40KB. It works fine when the attachment to SOAP is less than that. This is really absurd. Using Wireshark, it was determined that the communication failed at the TCP level. Because it is hard to change the receiver side, and it is relatively easier to change on the client side, we decided to alter the http transport layer that came with CXF.

To make things more complicated, our client uses Karaf as the OSGi impl.  I first followed the instruction of adding META-INF\services\javax.xml.ws.spi.Provider with the contents com.sun.xml.internal.ws.spi.ProviderImpl. But Karaf didn’t care that at all. Actually, Karaf throws a weird exception:

javax.xml.ws.spi.FactoryFinder$ConfigurationError: Provider org.apache.cxf.jaxws.spi.ProviderImpl not found

After many trials, the only thing that worked was to use dynamic System property setting at run time before the Webservice  client call.

String existingSpi = System.getProperty(“javax.xml.ws.spi.Provider”);

System.setProperty(“javax.xml.ws.spi.Provider”, “com.sun.xml.internal.ws.spi.ProviderImpl”); 

//  —JAX-WS code calls Webservices —

System.setProperty(“javax.xml.ws.spi.Provider”, existingSpi);

This successfully avoids CXF impl. to the default JAX-WS impl for the HTTP SPI provider implementation.

Note that there is really nothing wrong with CXF impl per se, but under the Karaf environment runtime, the above code allows us to alter the provider to make our client work with the server.

Cheers!

-Tony

java.lang.AbstractMethodError: javax.ws.rs.core.UriBuilder.uri

Hi, There:

This should be an easy one to figure out. But it took me a couple of days. :-((  When I set up my restful service using JAX-RS on Weblogic 10. I used Jersey reference implementation and defined the web.xml as:

<servlet>

<display-name>My Jersey Application</display-name>
<servlet-name>MyJerseyApp</servlet-name>
<servlet-class>com.sun.jersey.spi.container.servlet.ServletContainer</servlet-class>
<init-param>
<param-name>javax.ws.rs.Application</param-name>
<param-value>MyRestApplication</param-value>
</init-param>
<init-param>
<param-name>com.sun.jersey.config.property.packages</param-name>
<param-value>test.package</param-value>
</init-param>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>MyJerseyApp</servlet-name>
<url-pattern>/resources/*</url-pattern>
</servlet-mapping>

And my POM has

<dependency>
<groupId>org.glassfish.jersey.containers</groupId>
<!– if your container implements Servlet API older than 3.0, use “jersey-container-servlet-core” –>
<artifactId>jersey-container-servlet</artifactId>
<version>2.3</version>
</dependency>

BUT when I access the resources HelloWorld.java as restful impl class, I got this java.lang.AbstractMethodError: javax.ws.rs.core.UriBuilder.uri. It turned out I just need to include the dependency class for UriBuilderImpl. The easiest one is from CXF.

<dependency>
<groupId>org.apache.cxf</groupId>
<artifactId>cxf-bundle-jaxrs</artifactId>
<version>2.7.7</version>
</dependency>

Then the problem was gone!   I should have known that the java.lang.AbstractMethodError tells me it needed some actual Impl. class for the UriBuilder Abstract class. Duh!

Hope it helps anyone out there.

-TY

CXF Conduit setting for SSL and non-SSL client

Hi, there: Long time no see again. Spent a few days on some thing strange using CXF. I finally found out that there is some buggy issue with CXF Client. When hitting at WS on HTTPS/SSL, only cxf.xml will be effective. In details, the conduit client and authorization has to be configured in cxf.xml to make them work.

<http:conduit name=”https://.*”&gt;
<http:authorization>
<sec:UserName>userid1</sec:UserName>
<sec:Password>userpass1</sec:Password>
<sec:AuthorizationType>Basic</sec:AuthorizationType>
</http:authorization>
<http:client AutoRedirect=”false” Connection=”Keep-Alive”
AllowChunking=”true” ReceiveTimeout=”3600000″ ConnectionTimeout=”3600000″ />
</http:conduit>

But for NON-SSL request,  you need to configure them in code:

HTTPConduit http = (HTTPConduit) cl.getConduit();
HTTPClientPolicy httpClientPolicy = new HTTPClientPolicy();
httpClientPolicy.setAllowChunking(true);
httpClientPolicy.setConnectionTimeout(TIMEOUT);
httpClientPolicy.setReceiveTimeout(TIMEOUT);
httpClientPolicy.setAsyncExecuteTimeout(TIMEOUT);
http.setClient(httpClientPolicy);

AuthorizationPolicy authorizationPolicy = new AuthorizationPolicy();
authorizationPolicy.setUserName(testUserID);
authorizationPolicy.setPassword(testUserPass);
authorizationPolicy.setAuthorizationType(“Basic”);
http.setAuthorization(authorizationPolicy);

I suspect this is a bug in CXF (at least version 2.74).   I haven’t fully tested it to make cxf.xml to work with NON-SSL. But I am sure there is this bug that for HTTPS, only cxf.xml file can work.

Cheers!

-TY