la4j new feature

[yuronew]

Hi!

I was working on my educational project, and used la4j for some routine (Matrix and Vectors initialization, multiplying etc.)

I needed to solve system of linear equations, but surprisingly it was degenerate. I used SMART solver, but I got huge problem - some variables in result were near infinity.

So, I decided to implement one of approximate algorithms - for this purpose I took Nonlinear Conjugate Gradient method.

Now, I suggest you to take a look at this pull request. Here it is new package in la4j, org.la4j.opimization, which is designed same as org,la4j,linear. But with this type of solver user can find approximate solution with some accuracy, if it is possible. The bonus is that user can also search for solution for standard equation Ax=b, even if number of variables is greater then equations. I used this solver for 45x70 matrix, and got pretty good result.

Waiting for your response.

Regards,

Iurii Drozd.

[vkostyukov]

Hi Iurii @yuronew,

You've done a great job! This new functionality is something that is realy needed in la4j. But I have several concerns regarding naming/desing/code style/code performance/etc. So I would suggest to discuss all the issue here.

Terminology and Naming

I would start with naming or terminology. Probably, I am missing something, but seeing a Nonlinear word in a Linear Algebra package is a bit confusing. Your algorithm looks like a usuall/linear Conjugate Gradient method. I mean, we have to be very carefully with naming, since la4j is used by highly-qualified professionals in LA and Math in particular. Could you please point a proof-link where such method has a different name. At la4j we are working only with linear systems (output = input * N, linear dependency between input and output). This "Nonlinear ...." method is responsible for solving other systems (for example, output = input ^ 2, nonlinear dependency). So, we have to find a correct name first of all.

What else. We have to answer the question "What kind of problems are solved by such method?". As far as I know, this is just another interative method for solving determined (m=n) system of linear equations. According to the wikipedia, the coefficient matrix should be symmetric and positive defenit. So, I have a question. How did you solve an undetermined (m < n) linear system with such method? It looks like we miss something. And before starting to review implementation, I want to understand all these tihings. Hope, you will help me.

[yuronew]

Ok.

All necessary information you can find here http://en.wikipedia.org/wiki/Nonlinear_conjugate_gradient_method

For more explanations, you can take a look on this article http://goo.gl/jFyktV

The point is that user can use this method for any linear system, even if m > n.

[vkostyukov]

Well, I've looked thought the slides, which is actually about linear method. There is only one slide 3-22, which says that to make it works in non-linear system we have to perform some modifications. So, I'm not sure that this algorithm is a non-linear implementation. As I ca see, this is a simple linear conjugate gradient method. Actually, we can't use non-linear word in LinearAlgebra pacakge. It doesn't make sense, since these are difference arreas of study.

Also, I can't see the proof that this method can be used for undetermined systems (m < n). As far as I know, there is something that called "Minimum norm solution" is used for such systems. But you're saying that you used this method for 45x70 system. Could you please, provide more details about it (like input and output of the algorithm)?

I would like to see such method in la4j, but we have to rename it to something more convinient. And I'm currently trying to understand two things:

1. What king of systems can be solved with such method?
2. Is this method convergent?
3. How to name it properly and where to put (i.e., org.la4j.linear)?

[yuronew]

@vkostyukov
Ok. You can take a look on tests that are in my pull request. Firsts of them are from tests for Gaussian solving, and the last is test to optimize system 3x4.

1. It's not for solving, it's for optimization. We can apply this method to the systems that don't have solution. With this method we can find approximate solution, i.e. optimize system. The requirements are sleek - function ||A*x - b||^2 is be twice differentiable at the minimum.

2. Yes, it's convergent - but the number of steps required depends on function we are trying to optimize - the best case is N - the number of variables.

3. I don't know. These method is something to find the best approximate solution of system of linear equations. If solution exists, we will find it with great accuracy, otherwise we will find the best approximation.

[yuronew]

Hi @vkostyukov !
So what do you think about these new feature? I'm about to implement another one "optimizer" :)

[vkostyukov]

Hi @yuronew, I'm trying to clear all my knowledge about LA, in order to find a way to implement it carefully.

I know there is an "Problem of soling system of linear equations". I've never hear about "optimization problem" in terms of LA. There are such problems for functions (linke minimization/maximization) not for linear systems. Anyway, I would realy appreciate if you point me a right direction for search: "optimization problems in LA".

[yuronew]

Ok @vkostyukov , let me try.

I'm just named this set of methods "optimization", because we call them so at our classes. Btw, it's not an optimization of linear system. it's optimization of function. As far as I understand, la4j is a library for linear algebra, and optimization of functions is out of its scope. That's why I suggested such a trick - implement optimization algorithms for approximate solving of non-determined systems of linear equations. I think it can be useful in real tasks, at least i experienced problem when I used la4j - I couldn't find approximate solution.

Maybe, we can call this set of methods "approximation", or just make my method to be a part of "linear" package. But then we have to find out, how user can set needed epsilon, or number of steps.

I suggest you to run this new method on some test systems. You'll be surprised, how accurate it can be in some cases. Also, I can add system I mentioned, from my course work on system analysis. Part of it is here https://github.com/yuronew/system-analysis, but it lacks final commits. I can prepare it for you for tomorrow.

Thank you, waiting for your response.

[vkostyukov]

Hi Iurii @yuronew,

Please, don't think that I've forgotten about your PR. I believe we will work it out and these changes will be the part of 0.5.0 release.

Anyway, I'll come back with questions soon. Stay notified ;)