Notes on the Microsoft Hadoop/Sqoop connector

The Sqoop connector has been ported for both SQL Server and Oracle (ORAOOP), among others. However, both the SQL Server and Oracle connectors can prove problematical in operation. For instance, you can specify a query to extract a subset of data from SQL Server, but you cannot do this with Hive (you have to extract an entire table). This may mean an additional data preparation phase, where you prepare on Hive a table of just the data you are interested in, before pulling it into SQL Server with Sqoop. Sure, you can pull an entire table, but physical restrictions can make moving gigabytes of data a painfully slow process, which can be affected by network time outs, outages or transport limits. This pain is amplified if your OLTP and Hadoop platform are remote from each other. Even if they are on the same network, if you are moving hundreds of gigabytes of data around, you have to think how this will affect the rest of your network.

Remember that...