SQL Server 2016 Developer's Guide

The first technology that has been introduced in SQL Server 2016 to address the need for increased and improved security is Row Level Security (RLS). RLS provides the ability to control access to the rows in a table based on the user executing a query. With RLS it is possible to implement a filtering mechanism on any table in a database completely transparently to any external application or direct T-SQL access. The ability to implement such filtering without having to redesign a data access layer allows system administrators to control access to data at an even more granular level than before.

The fact that this control can be achieved without any application logic redesign makes this feature potentially even more attractive to certain use cases. RLS also makes it possible, in conjunction with the necessary auditing features, to lock down a SQL Server database so that even the traditional "god-mode" sysadmin cannot access the underlying data.

The second security feature that we will be covering is Dynamic Data Masking (DDM). DDM allows the system administrator to define column level data masking algorithms that prevent users from reading the sensitive content of columns, while still being able to query the rows themselves. This feature seems to have been initially aimed at allowing developers to work with a copy of production data without having the ability to actually see the underlying data. This can be particularly useful in environments where data protection laws are enforced (for example, credit card processing systems, medical record storage). The data masking occurs for unauthorized users at query runtime and does not affect the stored data of a table. This means that it is possible to mask a multi-terabyte database through a simple DDL statement, rather than resorting to the previous solution of physically masking the underlying data in the table we want to mask. The current implementation of DDM provides the ability to define a fixed set of functions to columns of a table, which will mask data when a masked table is queried. If a user has permission to view the masked data, then the masking function(s) are not run, whereas a user without those permissions will be provided with the data as seen through the defined masking functions.

The third major security feature to be introduced in SQL Server 2016 is Always Encrypted. Encryption with SQL Server was previously a (mainly) server-based solution. Databases were either protected with encryption at the database level (the entire database was encrypted) or at the column level (single columns had an encryption algorithm defined). While this encryption was and is fully functional and safe, crucial portions of the encryption process (for example, encryption certificates) are stored inside SQL Server. This effectively gave the owner of a SQL Server instance the potential ability to gain access to this encrypted data; if not directly, there was at least an increased surface area for a potential malicious access attempt. As more and more companies moved into hosted service and cloud solutions (for example, Microsoft Azure), the old encryption solutions no longer provided the required level of control and security. Always Encrypted was designed to bridge this security gap by removing the ability of an instance owner to gain access to the encryption components. The entirety of the encryption process was moved outside SQL Server and resides on the client-side. Previously, you could achieve a similar effect using a homebrew solution, but Always Encrypted provides a fully integrated encryption suite into both the .NET Framework and SQL Server. Whenever data is defined as requiring encryption, the data is encrypted within the .NET Framework and only sent to SQL Server after encryption has occurred. This means that a malicious user (or even system administrator) will only ever be able to access encrypted information should they attempt to query data stored via Always Encrypted.

This concludes the overview of the three main security enhancements inside SQL Server 2016. Microsoft has made some positive progress in this area. While no system is completely safe, and no single feature can provide an all-encompassing solution, each of these three features provide a further option in building up, or improving upon, any system's current security level. As mentioned for each feature, consult the dedicated chapter to explore how each feature functions and how they may be used in your environments.

The Query Store is possibly the biggest new engine feature to come with the release of SQL Server 2016. DBAs and developers should be more than familiar with the situation of a query behaving reliably for a long period, which suddenly changed into a slow-running, resource-killing monster query. Some readers may identify the cause of the issue being the phenomenon "parameter sniffing" or similarly "stale statistics". Either way, when troubleshooting why an unchanging query suddenly becomes slow, knowing the query execution plan(s) that SQL Server has created and used can be very helpful. A major issue when investigating these types of problems is the transient nature of query plans and their execution statistics. This is where Query Store comes into play; SQL Server collects and permanently stores statistics on query compilation and execution on a per database basis. This information is then persisted inside each database that has Query Store enabled, allowing a DBA or developer to investigate performance issues after the fact. It is even possible to perform query regression analysis, providing an insight into how query execution plans change over a longer timeframe. This sort of insight was previously only possible via hand-written solutions or third-party monitoring solutions, which may still not allow the same insights as the Query Store does.

When we are developing inside SQL Server, each developer creates a mental model of how data flows inside SQL Server. Microsoft has provided a multitude of ways to display this concept when working with query execution. The most obvious visual aid is the graphical execution plan. There are endless explanations in books, articles and training seminars which attempt to make reading these graphical representations easier. Depending upon how your mind works, these descriptions can help or hinder your ability to understand the data flow concepts: fully blocking iterators, pipeline iterators, semi-blocking iterators, nested loop joins, the list goes on. When we look at an actual graphical execution plan, we are seeing a representation of how SQL Server processed a query: which data retrieval methods were used, which join types were chosen to join multiple data sets, what sorting was required, and so on. However, this is a representation after the query has completed execution. Live Query Statistics offers us the ability to observe during query execution and identify how, when, and where data moves through the query plan. This live representation is a huge improvement in making the concepts behind query execution clearer and is a great tool to allow developers to better design their query and indexing strategies to improve query performance.

Microsoft has worked on their "Mobile First, Cloud First" strategy a lot in the past few years. We have seen a huge investment in Azure, their cloud offering, with the line between on-premises IT and cloud-based IT being continually blurred. The features being released in the newest products from Microsoft continue this approach and SQL Server is taking steps to bridge the divide between running SQL Server as a fully on-premises solution and storing/processing relational data in the cloud. One big step in achieving this approach is the new Stretch Database feature with SQL Server 2016. Stretch Database allows a DBA to categorize the data inside a database, defining which data is "hot" (frequently accessed data) and which is "cold" (infrequently accessed data). This categorization allows Stretch Database to then move the "cold" data out of the on-premises database and into Azure cloud storage. The segmentation of data remains transparent to any user/application that queries the data which now resides in two different locations. The idea behind this technology is to reduce storage requirements for the on-premises system by offloading large amounts of archive data onto cheaper, slower storage in the cloud. This reduction should then allow the smaller "hot" data to be placed on smaller capacity, higher performance storage. The benefit of Stretch Database is the fact that this separation of data requires no changes at the application or database query level. Stretch Database has been implemented to allow each company to also decide for themselves how data is defined as "hot" or "cold", providing maximum flexibility with minimal implementation overhead. This is a purely storage level change, which means the potential ROI of segmenting a database is quite large.

Many DBAs who support multiple third-party applications running on SQL Server experience the difficulty of setting up their SQL Server instances according to the application requirements or best practices. Many third-party applications have prerequisites that dictate how the actual instance of SQL Server must be configured. A common occurrence is a requirement of configuring the "Max Degree of Parallelism" to force only one CPU to be used for query execution. As this is an instance-wide setting, this can affect all other databases/applications in a multi-tenant SQL Server instance (which is generally the case). With Database Scoped Configuration in SQL Server 2016 several previously instance level settings have been moved to a database level configuration option. This greatly improves multi-tenant SQL Server instances, as the decision, for example, how many CPUs can be used for query execution can be made at the database level, rather than for the entire instance. This allows DBAs to host databases with differing CPU usage requirements on the same instance, rather than having to either impact the entire instance with a setting or be forced to run multiple instances of SQL Server and possibly incur higher licensing costs.

There are many instances where DBAs or developers are required to implement a change tracking solution, allowing future analysis or assessment of data changes for certain business entities. A readily accessible example is the change history on a customer account in a CRM system. The options for implementing such a change tracking system are varied and each option has strengths and weaknesses. One such implementation that has been widely adopted is the use of triggers to capture data changes and store historical values in an archive table. Regardless of the implementation chosen, it was often cumbersome to develop and maintain these solutions. One of the challenges was incorporating table structure changes in the table being tracked. It was equally challenging creating solutions to allow for querying both the base table and the archive table belonging to it. The intelligence of deciding whether to query the live and/or archive data can require some complex query logic.

With the advent of Temporal Tables, this entire process has been simplified for both developers and DBAs. It is now possible to activate this "change tracking" on a table and push changes into an archive table with a simple change to a table's structure. Querying the base table and including a temporal attribute to the query is also a simple T-SQL syntax addition. As such, it is now possible for a developer to submit temporal analysis queries and SQL Server takes care of splitting the query between the live and archive data and returning the data in a single result set.

Traditional data storage inside SQL Server has used the row-storage format, where the data for an entire row is stored together on the data pages inside the database. SQL Server 2012 introduced a new storage format: Columnstore. This format stores the data as columns rather than rows, combining the data from a single column and storing the data together on the data pages. This storage format provides the ability for massive compression of data, orders of magnitude better than traditional row-storage. Initially, only non-clustered columnstore indexes were possible. With SQL Server 2014 clustered columnstore indexes were introduced, expanding the usability of the feature greatly. Finally, with SQL Server 2016 updateable columnstore indexes and support for In-Memory columnstore indexes have been introduced. The potential performance improvements through these improvements are huge.

This concludes the section outlining the engine features implemented in SQL Server 2016. Through Microsoft's heavy move into cloud computing and their Azure offerings, they have increased the need to improve their internal systems for themselves. Microsoft is famous for their "dogfooding" approach to using their own software to run their own business, and Azure is arguably their largest foray into this area. The main improvements in the database engine have been fueled by the need to improve their own ability to continue offering Azure database solutions and provide features to allow databases of differing sizes and loads to be hosted together.

The last major improvements in the T-SQL language allowed for better processing of running totals and other similar window functions. This was already a boon and allowed developers to replace arcane cursors with high performance T-SQL. These improvements are never enough for the most performance-conscious developers among us, and as such there were still voices requesting further incorporation of the ANSI SQL standards into the T-SQL implementation.

Notable additions to the T-SQL syntax include the ability to finally split comma separated strings via a single function call STRING_SPLIT() instead of the previous "hacky" implementations using loops, functions, XML conversions or even the CLR.

The sensible opposing syntax for splitting strings is a function to aggregate values together: STRING_AGG() returns a set of values in a comma separated string. This replaces similarly "hacky" solutions using the XML data type or one of a multitude of looping solutions. Each improvement in the T-SQL language further extends the toolbox that we as developers possess in order to manipulate data inside SQL Server.

It is quite common to meet developers outside the Microsoft stack who look down on products released from Redmond. Web developers in particular have been critical of the access to the latest data exchange structures, or rather lack of it. JSON has become the de facto data exchange method for the application development world. It is similar in structure to the previous "cool-kid" XML, but for reasons beyond the scope of this book, JSON has overtaken XML in general programming projects and is the expected payload for application and database communications. Microsoft has included JSON as a possible data exchange data type in SQL Server 2016 and provided a set of functions to accompany the data type.

In-Memory OLTP (codename Hekaton) was introduced in SQL Server 2014. The promise of ultra-high performance data processing inside SQL Server was a major feature when SQL Server 2014 was released. As expected with a newly implemented feature, there were a wide range of limitations in the initial release and this prevented many customers from being able to adopt the technology. With SQL Server 2016 a great number of these limitations have been either raised to a higher threshold or completely removed. In-Memory OLTP has received the required maturity and extension in its feature set to make it viable for prime production deployment. Chapter 11, Introducing SQL Server In-Memory OLTP, of this book will show an introduction to In-Memory OLTP, explaining how the technology works under the hood and how the initial release of the feature works in SQL Server 2014. Chapter 12, In-Memory OLTP Improvements in SQL Server 2016, will build on the introduction and explain how the feature has matured and improved with the release of SQL Server 2016.

Accessing or managing data inside SQL Server and developing data solutions are two separate disciplines, each with their own specific focus on SQL Server. As such, Microsoft has created two different tools, each tailored towards the processes and facets of these disciplines.

SQL Server Management Studio (SSMS), as the name suggests, is the main management interface between DBAs/Developers and SQL Server. The studio was originally released with SQL Server 2005 as a replacement and consolidation of the old Query Analyzer and Enterprise Manager tools. As with any non-revenue generating software, SSMS received less attention over the years than the database engine, with limitations and missing tooling for many of the newer features in SQL Server. With SQL Server 2016 the focus inside Microsoft has been shifted and SSMS has been de-coupled from the release cycle of SQL Server itself. This decoupling allows both SSMS and SQL Server to be developed without having to wait for each other or for release windows. New releases of SSMS are created on top of more recent versions of Visual Studio and have seen almost monthly update releases since SQL Server 2016 was released to the market.

SQL Server Data Tools (SSDT) is also an application based on the Visual Studio framework. SSDT is focused on the application/data development discipline. SSDT is much more closely aligned with Visual Studio in its structure and the features offered. This focus includes the ability to create entire database projects and solution files, an easier integration into source control systems, the ability to connect projects into automated build processes, and generally offering a developer-centric development environment with a familiarity with Visual Studio. It is possible to design and create solutions in SSDT for SQL Server using the Relational Engine, Analysis Services, Integration Services, Reporting Services, and of course, for Azure SQL Database.

This concludes the overview of programming enhancements inside SQL Server 2016. The improvements outlined are all solid evolutionary steps in their respective areas. New features are very welcome and allow us to achieve more, while requiring less effort on our side. The In-Memory OLTP enhancements are especially positive, as they now expand on the groundwork laid down in the release of SQL Server 2014. Read the respective chapters to gain a deeper insight into how these enhancements can help you.

Data analytics has been the hottest topic in IT for the past few years, with new niches being crowned as the pinnacle of information science almost as fast as technology can progress. However, IT does have a few resolute classics that have stood the test of time and are still in wide use. SQL (in its many permutations) is a language we are well aware of in the SQL Server world. Another such language is the succinctly titled R. The R language is a data mining, machine learning and statistical analysis language that has existed since 1993. Many professionals with titles such as data scientists, data analyst, or statistician have been using the R language and tools that belong in that domain ever since. Microsoft has identified that, although they may want all the world's data inside SQL Server, this is just not feasible or sensible. External data sources and languages such as R exist and need to be accessible in an integrated manner.

For this to work, Microsoft made the decision to purchase Revolution Analytics (a commercial entity producing the forked Revolution R) in 2015, which made it possible for them to integrate the language and server process into SQL Server 2016. This integration allows a normal T-SQL developer to interact with the extremely powerful R service in a native manner and allow more advanced data analysis to be performed on their data.