Performance Tuning Oracle BPEL

First, there are many sites that list some psuedo random steps that are related to performance tuning, but very few indicate why you should be doing them. While I can't answer why for all of them, I can provide the benefit of experience at least.

Threading Configuration

Threading has dramatically changed from 10.1.3.3 to 10.1.3.4. Depending on which version you're running, read the appropriate section below.




Prior to 10.1.3.4 Threading

First, it's important to understand how threads are used within BPEL. Basically, there is one main thread pool configured by the message-driven bean named WorkerBean. This MDB controls how many threads are available to ALL BPEL domains, and out of the box is set to 40 ReceiverThreads. Each BPEL domain has a single setting that controls how many of these it can allocate for processing instances of BPEL processes, called the dspInvokeMaxAlloc setting. Out of the box each domain believes it can allocate 100 instances of WorkerBeans, which obviously isn't good since there are only 40 for the entire OC4J BPEL instance. Not sure why this is how it comes configured, but it really needs to be changed to avoid BPEL domains instantiating more instances that

they can truly handle and thereby causing excessive timeouts.

Second, there are 3 main usages of WorkerBeans by a BPEL domain:



1. Process New Instance Requests - e.g. when an adapter initiates a BPEL process
   


2. Process Activities - e.g. when BPEL is actively running a BPEL process instance
   


3. Handle Engine Requests - e.g. callbacks for asynchronous processes


The first 2 usages are throttled by another BPEL domain setting named dspInvokeAllocFactor, which is a basic ratio of how many WorkerBeans would be used to process new instances versus activity requests. Out of the box it is set to 0.4, or roughly 40% of these MDBs can be used for new instances versus activities. It is important to note that this configuration does not control engine requests like callbacks. From what I can tell, internally Oracle reserves a percentage of WorkerBeans to handle engine requests separate from the ones available for new instances and activity requests.


10.1.3.4 Threading


Starting in 10.1.3.4, Oracle has revised their threading model. The WorkerBeans have disappeared, as have the domain-level properties dspInvokeMaxAlloc and dspInvokeAllocFactor. Instead, they now use domain properties named dspEngineThreads, dspSystemThreads and dspInvokeThreads. These properties are clearly described in the BPEL Console/Control under Configuration for each BPEL domain. For my recent project, which is almost exclusively asynchronous BPEL processes, the dspEngineThreads were the primary bottlenecks until we found the number of threads that gave us the throughput we needed without causing backlogged engine queues and timeouts in our Pick OnAlarm activities.


Similar to the prior versions, it is important to analyze the number of concurrent asynch and synch processes you expect to have at max volume so you can configure the appropriate number of dehydration database connections, as described below.





Number of Quartz Threads(per BPEL domain)

One issue we ran into was not seeing some OnAlarm branches from Pick activities being executed, even though it was long past the timeout time. This can also happen with Wait activities in times of volume. If your project uses an abundance of either activity to handle timeouts, you should verify the number of Quartz threads is configured appropriately for your BPEL domain. Rather than rewrite what others have described well, here's a helpful link to another blog which describes how to do exactly this: Mihai Munteanu's Blog site.




Database connection Configuration







Configure Database Adapter to use DB Connection Pool

One of the biggest issues when building BPEL processes that use the database adapter is understanding how your BPEL process actually gets a database connection, and making sure the server is configured to use a database connection pool.


The image to the right is referenced from the Oracle Application Server Adapters for Files, FTP, Databases, and Enterprise Messaging User's Guide, but shows how your BPEL process gets a database connection. While JDeveloper is really trying to be helpful by adding a bunch of mcf.* properties to your database adapter partner link so that a connection can be made at runtime to the database you are referencing from within JDeveloper at design time, it really can cause unexpected issues for those that don't understand how these database connections are obtained at runtime. My suggestion is to remove all mcf.* properties from your database adapter partner link WSDL files, which will force your process to either find the connection using the DbAdapter configuration on the server or fail with an error during deployment or runtime that will show you in the log files that it couldn't obtain a connection.

What you'll notice in your WSDL file for the database call is a location attribute with a setting like "eis/DB/...". This refers to a connection factory defined in the database adapter oc4j-ra.xml referenced in the figure above, which then should reference a JDBC datasource that uses a connection pool. So to get the ball started, a JDBC connection pool must be created within ASConsole/Enterprise Manager. Refer to the Oracle OC4J Services Guide for information on how to create a connection pool and data source. Make sure you select the correct SOA OC4J instance on a standalone application server deployment when following their instructions, as they assume you installed SOA into the Home OC4J and often people use another one named OC4J_SOA instead. Also, if using a clustered application server topology, you should have a group created that just contains the SOA OC4J instances in the cluster, and you can access the JDBC Resources by selecting the group name, then Administration, then JDBC Resources and your changes will affect the entire cluster at once.

After you have the connection pool created, and the datasource (XA preferably for connections that will insert/update/delete) created that references it, you can configure the DbAdapter for your process to use at runtime. From the ASConsole/Enterprise Manager, you can configure the DbAdapter by navigating to each SOA OC4J instance (if clustered you would need to access more than one), then selecting Applications->Default (application)->DbAdapter->Connection Factories. From here you can create a new Connection Factory with the same "eis/DB/..." JNDI path that then refers to one or more datasources. Note that JDeveloper names the "..." portion according to the name you created locally as the connection name, so if there are multiple developers involved this should be standardized so there aren't processes looking for different JNDI paths at runtime to the same database as the same user. A couple of points to know here. First, don't use the connection string property. Second, assuming you created an XA datasource, enter the JNDI to the datasource in the XA Datasource field. The documentation says you can also create a separate datasource that's not XA for select operations only, and populate this in the Connection Factory Datasource field, but in practice I've only needed to use XA datasources.


Change the Dehydration Database Driver


When BPEL Process Manager is installed (or upgraded often times, as in 10.1.3.4) in Oracle Application Server, it configured its connection pool using the OracleDriver JDBC class instead of the OracleDataSource class. The latter one provides better fine-grained control over connection pool options like validating the connection, and in my projects we have replaced the default pool with a new one that uses the OracleDataSource.


Configure Dehydration Database Connection Size

Also, you should configure the database connection pool used by BPEL for accessing the dehydration store to allow at least as many connections as WorkBeans/Engine Threads (and maybe InvokerBeans too), so in times of load each thread can immediately dehydrate as necessary. Even if you decide to shave some performance by not dehydrating your processes and instead process instances only in memory, in cases of faults BPEL should dehydrate failed instances to the database.


Tune the Dehydration Database

It cannot be overstated that it will be important to add indexes to the BPEL dehydration database, as well as the initial settings configuration mentioned in the Oracle App Server Performance Guide.




Timeout Settings


If you find some of your instances seem to never return in your asynchronous pick OnAlarm activities, or you see errors in the logs like


An exception occurred during transaction completion:; nested exception is:
javax.transaction.RollbackException: Timed out


then you should configure the transaction timeouts as described in the Oracle BPEL Process Manager Developer's Guide -Troubleshooting and Workarounds section.

These are standard tuning options I've found useful in my projects. Stay tuned for BPEL process tuning steps in a subsequent BLOG.