GSoC: First steps...

[JulienPeloton]

The project will be divided in 3 parts:

• Find alternative to JTS (no 3D support): new dev or existing package?
• Upgrade 2DRDD to 3DRDD (circle -> sphere, parallelogram -> parallelepiped, and so on).
• Upgrade 2D methods for partitioning/join/query/indexing to 3D.

[JulienPeloton]

Week 1:

• A few shapes (Shape3D, Point3D, Sphere) are ready (see Geometry objects #8 and Add simple Point3D + Sphere #9).
• One spatial partitioning of the space is there: linear onion space (see Partitioning using Healpix: the onion space #11 and First working version of Point3D RDD! #12).
• One 3D RDD is available (see First working version of Point3D RDD! #12).

A few TODO for week 2:

• Keep developing partitioners. TREE in 3D? @mayurdb ?
• Keep developing 3D RDD. I need to re-organize and make more general Shape3DRDD. Especially spatialPartitioning in Point3DRDD must be generalized and incorporated in Shape3DRDD.
• Remove rdd as input arg of partition[T<:Shape3D] and benchmark it.
• Need to add direct loading of CSV, FITS, etc. in addition to handling direct RDD[T] as it is done now.

[JulienPeloton]

Week 2:

• Shape3DRDD has been re-organized to be more generic with respect to partitioner (see Re-organisation of the way data is partitioned #15). Future partitioner would just have to provide an implementation of the partitioning, and the partition method in Shape3DRDD would do the job.
• Add healpix partitioning for shells in the onion partitioning (see f800e83).
• Add FITS reader (see Re-organisation of the way data is partitioned #15).
• Add CSV reader and a file test (see Add CSV reader + CSV test file #18). a script to convert FITS to CSV has been added, to ease the testing.
• Add example and visualisation tools (see Visualisation #19). We provide a notebook example (Jupyter notebook + Toree kernel).
• ...?

A few TODO for week 3:

• Tree partitioning
• Join/query methods for the onion partitioning.
• ...?

[JulienPeloton]

Week 3:

• Add replacement for JTS (see Envelope #22), which includes ShellEnvelope and BoxEnvelope. This also includes (Add Envelope/Shell/Box classes + Force Point3D to declare type of data #23):
  • Force Point3D to declare the type of data, that is spherical (r, theta, phi) or cartesian. It was optional before.
  • Integrate the development made into the Onion partitioning. Now it uses Shells instead of Sphere to partition the space.
• Add mechanisms to perform the cross-match of 2 data sets. (see Add new objects and routines to perform the cross-match of 2 data sets. #24)
• Add HOWTO notebooks.

A few TODO for week 4:

• Octree (see mayurdb@cfd20b3)
• Treatment of objects at the border of partitions (see Overlapping shells for OnionPartitioning #25)
• Multi-resolution algorithm for cross-match (see Multi-resolution search #26)

[JulienPeloton]

Week 4:

• Octree implementation (see Octree #30).
• Add very simple Kryo registration for spark3D classes. Probably need more (see Kryo #31, Set a more efficient serializer #28).
• Major README updates, and v0.1.0 release (see Readme update #33).
• Add scripts to generate test data (see Add scripts to generate test data #34).
• A few fixes: fix: added a break when an appropriate partition is found #27, Write the partitioner in the spirit of Scala (no more Java-like) #29, Fix typo when generating files with sphere: radius was too big! #35.

A few TODO for week 5:

• Treatment of objects at the border of partitions (see Overlapping shells for OnionPartitioning #25)
• Multi-resolution algorithm for cross-match (see Multi-resolution search #26)
• Create Sphere3DRDD class to simplify the use of octree.
• Tackling large shuffle? Tackling large shuffle? #32
• Implement RangeQuery methods (to find neighbours).