Chapter 11. Asynchronous Processing and Big Data Volumes

This chapter covers two important aspects of developing an enterprise-level application on the Force.com platform. While asynchronous processing and Big Data volumes are not necessarily linked, the greater governor limits and processing power of the platform are mostly to be found in the async processing mode. However, making sure that your interactive user experience is still responsive when querying large volumes of data is also very important.

In this chapter, we will first review how to ensure SOQL queries are as performant as possible by understanding how to profile queries and make use of the standard and custom indexes. This will benefit both interactive and batch (or async) processes in your application as well as how they leverage native platform features such as reporting.

We will then take a look at the options, best practices, design, and usability considerations to move your application processing into the async mode, once...

Creating test data for volume testing

Throughout this chapter, we are going to use the RaceData__c object. We will be extending it with additional fields and running a script to populate it with some test volume data, before exploring the different ways of accessing it and processing the data held within it.

In the Formula1 racing world, a huge amount of data is captured from the drivers and the cars during the race. This information is used to perform real-time or post race analysis in order to lead to further improvements or troubleshoot issues. Because of the huge volumes of data involved, the FormulaForce application considers ways to summarize this data such that it can be removed and later reloaded easily if required.

The following functionalities will be added to the FormulaForce application to process this information and help demonstrate the Force.com features described in this chapter. The Apex classes and Visualforce page for this functionality is included in the sample code for...

Indexes, being selective, and query optimization

In this section, we will review when system and custom indexes maintained by the Force.com platform are used to make queries more performant and, once larger query results are returned, the ways in which they can be most effectively consumed by your Apex logic.

Standard and custom indexes

As with other databases, Force.com maintains database indexes as record data is manipulated to ensure that, when data is queried, such indexes can be used to improve query performance. Due to the design of the Force.com multitenant platform, it has its own database index implementation (instead of using the underlying Oracle database indexes) that considers the needs of each tenant. By default, it maintains standard indexes for the following fields:

• ID

• Name

• OwnerId

• CreateDate

• CreatedById

• LastModifiedDate

• LastModifiedById

• SystemModStamp

• RecordType

• Any Master Detail fields

• Any Lookup fields

• Any fields marked as Unique

• Any fields marked as External Id

Once your application...

Skinny tables

Internally, Salesforce stores Standard Object's field data and custom field data for a given record in two separate physical Oracle tables. So, while you don't see it, when you execute a SOQL query to return a mixture of both standard and custom fields, this internally requires an Oracle SQL query, which requires an internal database join to be made.

If Salesforce Support determines that avoiding this join would speed up a given set of queries, they can create a skinny table. Such a table is not visible to the Force.com developer and is kept in sync with your object records automatically by the platform, including any standard or custom indexes.

A skinny table can contain commonly used standard and custom fields that you, the subscriber, and Salesforce deem appropriate, all in the one Oracle table, thus, it avoids the join (all at the internal cost of duplicating the record data). If a SOQL, Report, or List View query utilizes the fields or a subset, the platform will automatically...

Asynchronous execution contexts

Salesforce is committed to ensuring that the interactive user experience (browser or mobile response times) of your applications and that of its own is as optimal as possible. In a multitenant environment, it uses the governors as one way to manage this. The other approach is to provide a means for you to move the processing of code from the foreground (interactive) into the background (async mode). This section of the chapter discusses the design considerations, implementation options, and benefits available in this context.

As the code running in the background is, by definition, not holding up the response times of the pages the users are using, Salesforce can and does throttle when and how often async processes execute, depending on the load on the servers at the time. For this reason, it currently does not provide an SLA on exactly when an async piece of code will run or guarantee exactly when the Apex Schedule code will run at the allotted time. This...

Volume testing

In this chapter, we have used the Apex code to generate additional data to explore how the platform applies indexes and the use of Batch Apex. It is important to always perform some volume testing even if it is just with at least 200 records to ensure that your Triggers are bulkified. Testing with more than this gives you a better idea of other limitations in your software search, such as query performance and any Visualforce scalability issues.

One of the biggest pitfalls with volume testing is the quality of the test data. It is not often that easy to get hold of actual customer data, so we must emulate the spread of information by varying field values amongst the records to properly simulate how the software will not only behave under load with more records, but under load with a different dispersion of values. This will tease out further bugs that might only manifest if the data values change within a batch of records. For example, the Apex script used in this chapter could...

Summary

In this chapter, we have learned that understanding not just the volume of records, but also how data within them is dispersed can affect the need for indexes to ensure that queries perform without having to resort to expensive table scans or, in the case of Apex Trigger, runtime exceptions. Ensuring that you and your customers understand the best way to apply indexes is critical to both interactive and batch performance.

Asynchronous execution contexts open up extended and flexible control over the governors available in the interactive context, but need to be designed from the beginning with careful consideration for the user experience, such that the user is always aware of what is and is not going on in the background. Messaging is the key when there is no immediate user interface to relay errors and resolve them. Don't depend on the Apex Jobs screen; instead, try to contextualize error information through custom log objects related to key records in your application that drive...