Understanding Large Data Volumes

While the Salesforce platform can cope with large amounts of data, some considerations apply to larger/massive amounts of data (referred to as Large Data Volumes (LDV)) and how performance on the platform is affected. This chapter covers LDV considerations and mitigations, as well as scalable data model design and data archiving strategies.

In this chapter, we will cover the following topics:

• Designing a scalable data model
• LDV performance mitigation strategies
• Data archiving strategies

Unlike traditional applications that utilize a database, Salesforce stores all data in a few, large database tables. Therefore, traditional performance tuning techniques associated with databases don’t necessarily apply to the Salesforce platform. Instead, we, as data architects, must design our Salesforce implementations to handle large amounts of data. This is best achieved by understanding LDVs, their impact on Salesforce performance...

Designing a scalable data model

As Salesforce implementations grow in size and complexity, so does the volume of data. Salesforce, being a multi-tenant architecture, handles the scaling up automatically, but as the volume of data grows, the processing time for certain operations increases too.

Typically, two areas are affected by different data architectures or configurations on the Salesforce platform:

• Loading or updating large amounts of records. This can be through the UI (directly) or with one or more integrations.
• Extracting data, be it through reports or other views into the data or querying the data.

Optimizing the data model generally involves doing the following:

• Only hosting data that truly needs to reside on the Salesforce platform based on business purpose and intent
• Deferring or temporarily disabling sharing change processing and other business rule logic when performing certain data operations
• Choosing the best (most efficient) operation...

LDV performance mitigation strategies

There are several tools available to us as data architects when it comes to working with data and keeping things running as optimally as possible. Of course, we should first question what data needs to reside in Salesforce. How often should we consider moving data off-platform to reduce performance impacts due to having large amounts of data available? Typically, data of a certain age (determined by business requirements or regulatory requirements in some instances) should be archived regularly to ensure that users are only interacting with the data that makes sense to be hosted on the Salesforce platform.

With data that may need to reside on Salesforce, some other techniques are available that can improve performance:

• Custom indexes
• Skinny tables
• Selective filter conditions
• Divisions

Let’s take a look at them.

Custom indexes

To speed up query performance, Salesforce supports the creation of custom indexes...

Data archiving strategies

Just as we have data on Salesforce for our users, there are situations where data needs to be archived off the platform. This may be due to various reasons, such as regulatory compliance (where certain data needs to be retained), or to keep the optimum amount of data in the platform (such as only the data that is being used being kept on the platform, and automatically archiving data over a certain age). Luckily, various options are available for archiving Salesforce data, such as using on-platform solutions such as big objects or storing data off-platform in an external system or data warehouse. We’ll take a look at these options in the following subsections.

Big objects

As covered in Chapter 2, Data Modeling and Database Design, big objects are used to store and manage huge amounts of data (up to 1 million records by default, though this can be scaled up at an additional cost to tens of millions or even billions of records).

Big objects provide...

Summary

In this chapter, we dug deep into the data architecture and its potential pitfalls and mitigation strategies. Understanding why data skew happens helps us design better parent/child record ownership strategies. Also, understanding the way Salesforce uses indexes ensures that we can create reports, list views, and build queries that work with the constraints of the multi-tenant architecture nature of the platform, not against them. Due to this, we looked at LDV issues and mitigation strategies, understanding how concepts such as selective filter conditions and skinny tables can be used to ensure we can work with our large amounts of data effectively.

Next, we turned our attention to data archiving strategies and the various options we have at our disposal, ensuring that we are keeping to regulatory requirements for data retention and archival (if appropriate) and ensuring that we only use the data that is relevant to our users.

In Chapter 7, Data Migration, we’re...

Practice questions

Test your knowledge of the topics covered in this chapter by answering the following questions:

1. What does LDV stand for?
2. Which type of Salesforce object is used to store 1 million or more records?
3. One account that contains over 10,000 child records is known as what?
4. More than 10,000 records looking up to a single parent record is known as what?
5. Utilizing data hosted on an external system is a technique known as what?
6. What can be used to partition data and reduce the number of records returned by SOQL queries and reports?
7. What can be used to avoid joins within queries to data held within a single object and speed up read-only operations?
8. More than 10,000 child records belonging to the same parent record is known as what?
9. Specify the type of index that’s used in the following scenarios:
   • When a SOQL query is executed against object data where that object contains 400,000 records, and the filter matches 40,000 or fewer records...

Further reading

See the following resources for more information on what was covered in this chapter:

• Salesforce Trust: https://trust.salesforce.com
• Search Architecture: https://developer.salesforce.com/docs/atlas.en-us.salesforce_large_data_volumes_bp.meta/salesforce_large_data_volumes_bp/ldv_deployments_concepts_search_architecture.htm
• Indexes: https://developer.salesforce.com/docs/atlas.en-us.salesforce_large_data_volumes_bp.meta/salesforce_large_data_volumes_bp/ldv_deployments_infrastructure_indexes.htm
• Skinny tables: https://developer.salesforce.com/docs/atlas.en-us.salesforce_large_data_volumes_bp.meta/salesforce_large_data_volumes_bp/ldv_deployments_infrastructure_skinny_tables.htm