Development of Julia module and R-package for SR1 and sum-modeling facilities of Lancelot (and Galahad)

[JimEBlevins]

This is a request for the development of interfaces for R and Julia, for the needs of statisticians and of researchers in machine learning and stochastic optimization.

R system for statistics

Statisticians continue to use [L-]BFGS for computing approximate Hessians (which are used for limiting standard errors, p-values, confidence intervals, and tests).
Perhaps you could collaborate with a computational statistician (or the R Foundation) to produce an R-package interface to Galahad, particularly for the SR1 unconstrained optimization facilities of Lancelot.

Julia for sum problems

Statisticians also are concerned with minimizing sums of functions, for solving estimating equations and for M-estimation (least squares, maximum likelihood for i.i.d. errors).
Similar problems arise in stochastic optimization (e.g., sample average approximation) and in machine learning, where stochastic [quasi- or sub-] gradient methods are popular.

For computational research in statistics, stochastic optimization, and machine learning, the sum-modeling facilities of Lancelot (and presumably Galahad) would be useful.

A Julia interface to the sum-modeling facilities of Lancelot (and presumably Galahad) would be great.

(It might be possible to interest Professor Christoph Helmberg in collaboration, e.g., with developing a Julia (or R) interface to the sum-modeling facilities of his ConicBundle code for nonsmooth convex-minimization, which would be useful for computing the spatial median, for example.)

[nimgould]

Dear Jim

Thanks for all of this. We are currently using the C interfaces we developed last year as building blocks onwards into python and julia (not R, but that is possible, it simply doesn't cross our horizon and we have no experience at all). There is a caveat, namely that Lancelot is the one package from Galahad that we have not translated as yet, partially because we didn't expect much demand, and also because the partially-separable interface is significantly more complicated than the more-traditional function-gradient-(maybe )Hessian one that most users expect.

One issue though, for machine learning I might be wrong but I thought that each of the sub functions in the sum involved all parameters, in which case the individual Hessians are dense. This doesn't fit well with partial separability, where the underlying assumption is that the Hessians are sparse, or at least structured low rank. Fine if there aren't too many parameters, but difficult/impossible for machine learning with millions of parameters. As I said, maybe this is just a misunderstanding on my part.

The python/julia interfaces are taking all of our time at present, but we will keep this in mind once we have some free time ... and yes, finding interested, knowledgeable collaborators is always helpful.

Nick

[JimEBlevins]

Hi Nick!

It is great to hear about the work towards a Julia interface, which would be more accessible to R-users. R-contributors could develop explicit interfaces using the existing Julia--R interfaces [link to ongoing StackOverflow discussion with leading contributors].

Regarding partial separability:

I agree that statistical uses are negligible compared to the FEM applications of partial separability. I found only 38 statistical citations that are concerned with Hessians (Fisher information matrix) and mention partial separability. Some of these papers mention functional-MRI statistics (which may be related to FEM).

Other statistical applications include spline-based density estimation, following J. W. Schmidt.

Validation and experimentation

For dense problems (regardless of additivity), your code would be useful as a benchmark for evaluating other iterative-methods; such validation of statistical software is important for FDA and EMA submissions.

A second interest is to use any additivity-facilities for experimentation with, e.g., stochastic [quasi-]gradient methods for interesting problems (rather than impressing people with the number of variables or the size of the data). A Julia-laboratory (developed with an eye towards Turing and JuMP) would be very useful to researchers in machine learning.

Thanks

Above, I mentioned the spatial-median problem. It is fitting to note that the spatial median problem is related to the Fermat--Weber location problems, to which the late Professor Andrew R. Conn made many contributions (besides partial separbility, Lancelot, Galahad, etc.). I am very grateful for your team's work.