In this recipe, we will use the UCI admission dataset again so can demonstrate Spark's RDD-based logistic regression solution, LogisticRegressionWithLBFGS(), for an extremely large number of parameters that are present in certain types of ML problem.
We recommend L-BFGS for very large variable space since the Hessian matrix of second derivatives can be approximated using updates. If you have an ML problem with millions or billions of parameters, we recommend deep learning techniques.
- We use the admission dataset from UCLA IDRE. You can download the entire dataset from the following URLs:
- For home page go through the http://www.ats.ucla.edu/stat/link.
- For the data file go through the https://stats.idre.ucla.edu/stat/data/binary.csvlink.
The dataset comprises four columns, with the first column being the dependent variable (that is, the label - whether the student was admitted or not) and the next three columns being the...
