Merge pull request #83 from astrolabsoftware/0.1.5

0.1.5

.gitignore

@@ -3,3 +3,6 @@ metastore_db
 derby.log
 old/
 .ipynb_checkpoints/
+.DS_Store
+.log
+misc

CHANGELOG.md

@@ -1,3 +1,9 @@
+## 0.1.5
+
+- Allow user to cache the rawRDD while loading data to speed-up re-partitioning. ([PR](https://github.com/astrolabsoftware/spark3D/pull/81))
+- Script to benchmark the partitioning ([PR](https://github.com/astrolabsoftware/spark3D/pull/72))
+- Several minor fix ([PR](https://github.com/astrolabsoftware/spark3D/pull/71), [PR](https://github.com/astrolabsoftware/spark3D/pull/76), [PR](https://github.com/astrolabsoftware/spark3D/pull/80))
+
 ## 0.1.4
 
 - Unify the IO: One constructor to rule them all! ([PR](https://github.com/astrolabsoftware/spark3D/pull/69))

README.md

@@ -9,9 +9,9 @@
 - [05/2018] **GSoC 2018**: spark3D has been selected to the Google Summer of Code (GSoC) 2018. Congratulation to [@mayurdb](https://github.com/mayurdb) who will work on the project this year!
 - [06/2018] **Release**: version 0.1.0, 0.1.1
 - [07/2018] **New location**: spark3D is an official project of [AstroLab Software](https://astrolabsoftware.github.io/)!
-- [07/2018] **Release**: version 0.1.3, 0.1.4
+- [07/2018] **Release**: version 0.1.3, 0.1.4, 0.1.5
 
-<p align="center"><img width="500" src="https://github.com/astrolabsoftware/spark3D/raw/master/pic/spark3d_lib_0.1.4.png"/>
+<p align="center"><img width="500" src="https://github.com/astrolabsoftware/spark3D/raw/master/pic/spark3d_lib_0.1.5.png"/>
 </p>
 
 ## Installation and tutorials

build.sbt

@@ -19,7 +19,7 @@ import xerial.sbt.Sonatype._
 lazy val root = (project in file(".")).
  settings(
    inThisBuild(List(
-     version      := "0.1.4"
+     version      := "0.1.5"
    )),
    // Name of the application
    name := "spark3D",

docs/01_installation.md

@@ -20,7 +20,7 @@ another version, feel free to contact us. In addition to Spark, the library has
 You can link spark3D to your project (either `spark-shell` or `spark-submit`) by specifying the coordinates:
 
 ```bash
-toto:~$ spark-submit --packages "com.github.astrolabsoftware:spark3d_2.11:0.1.4" <...>
+toto:~$ spark-submit --packages "com.github.astrolabsoftware:spark3d_2.11:0.1.5" <...>
 ```
 
 It might not contain the latest features though (see *Building from source*).
@@ -69,7 +69,7 @@ result on the screen, plus details of the coverage at
 First produce a jar of the spark3D library, and then launch a spark-shell by specifying the external dependencies:
 
 ```bash
-toto:~$ JARS="target/scala-2.11/spark3d_2.11-0.1.4.jar,lib/jhealpix.jar"
+toto:~$ JARS="target/scala-2.11/spark3d_2.11-0.1.5.jar,lib/jhealpix.jar"
 toto:~$ PACKAGES="com.github.astrolabsoftware:spark-fits_2.11:0.6.0"
 toto:~$ spark-shell --jars $JARS --packages $PACKAGES
 ```
@@ -83,7 +83,7 @@ scala> // etc...
 Note that if you make a fat jar (that is building with `sbt assembly` and not `sbt package`), you do not need to specify external dependencies as they are already included in the resulting jar:
 
 ```bash
-toto:~$ FATJARS="target/scala-2.11/spark3D-assembly-0.1.4.jar"
+toto:~$ FATJARS="target/scala-2.11/spark3D-assembly-0.1.5.jar"
 toto:~$ spark-shell --jars $FATJARS
 ```
 

docs/_config.yml

@@ -18,7 +18,7 @@ locale                   : "en-US"
 title                    : "spark3D"
 title_separator          : "-"
 name                     : "spark3D"
-description              : "Apache Spark extension for processing large-scale 3D data sets, such as astrophysical or high energy physics data."
+description              : "Spark extension for processing large-scale 3D data sets: Astrophysics, High Energy Physics, Meteorology, ..."
 url                      : https://astrolabsoftware.github.io
 baseurl                  : /spark3D
 repository               : "astrolabsoftware/spark3D"

docs/_pages/home.md

@@ -8,7 +8,7 @@ header:
   cta_url: "/docs/installation/"
   caption:
 intro:
-  - excerpt: '<p><font size="6">Spark extension for processing large-scale 3D data sets: Astrophysics, High Energy Physics, Meteorology, ...</font></p><br /><a href="https://github.com/astrolabsoftware/spark3D/releases/tag/0.1.4">Latest release v0.1.4</a>'
+  - excerpt: '<p><font size="6">Spark extension for processing large-scale 3D data sets: Astrophysics, High Energy Physics, Meteorology, ...</font></p><br /><a href="https://github.com/astrolabsoftware/spark3D/releases/tag/0.1.5">Latest release v0.1.5</a>'
 excerpt: '{::nomarkdown}<iframe style="display: inline-block;" src="https://ghbtns.com/github-btn.html?user=astrolabsoftware&repo=spark3D&type=star&count=true&size=large" frameborder="0" scrolling="0" width="160px" height="30px"></iframe> <iframe style="display: inline-block;" src="https://ghbtns.com/github-btn.html?user=astrolabsoftware&repo=spark3D&type=fork&count=true&size=large" frameborder="0" scrolling="0" width="158px" height="30px"></iframe>{:/nomarkdown}'
 feature_row:
   - image_path:

docs/about.md

@@ -43,7 +43,7 @@ Several goals have to be undertaken in this project:
 
 ## Current structure
 
-<p align="center"><img width="500" src="https://github.com/astrolabsoftware/spark3D/raw/master/pic/spark3d_lib_0.1.4.png"/>
+<p align="center"><img width="500" src="https://github.com/astrolabsoftware/spark3D/raw/master/pic/spark3d_lib_0.1.5.png"/>
 </p>
 
 ## Support

examples/jupyter/CrossMatch.ipynb

@@ -66,48 +66,47 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
-   "metadata": {
-    "collapsed": true
-   },
+   "execution_count": 1,
+   "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
       "Marking com.github.astrolabsoftware:spark-fits_2.11:0.6.0 for download\n",
       "Preparing to fetch from:\n",
-      "-> file:/var/folders/my/lfvl285927q2hzk545f39sy40000gn/T/toree_add_deps7614799531666607290/\n",
+      "-> file:/var/folders/my/lfvl285927q2hzk545f39sy40000gn/T/toree_add_deps5098665649624919110/\n",
       "-> https://repo1.maven.org/maven2\n",
-      "-> New file at /var/folders/my/lfvl285927q2hzk545f39sy40000gn/T/toree_add_deps7614799531666607290/https/repo1.maven.org/maven2/com/github/astrolabsoftware/spark-fits_2.11/0.6.0/spark-fits_2.11-0.6.0.jar\n",
+      "-> New file at /var/folders/my/lfvl285927q2hzk545f39sy40000gn/T/toree_add_deps5098665649624919110/https/repo1.maven.org/maven2/com/github/astrolabsoftware/spark-fits_2.11/0.6.0/spark-fits_2.11-0.6.0.jar\n",
       "Marking com.github.haifengl:smile-plot:1.5.1 for download\n",
       "Preparing to fetch from:\n",
-      "-> file:/var/folders/my/lfvl285927q2hzk545f39sy40000gn/T/toree_add_deps7614799531666607290/\n",
+      "-> file:/var/folders/my/lfvl285927q2hzk545f39sy40000gn/T/toree_add_deps5098665649624919110/\n",
       "-> https://repo1.maven.org/maven2\n",
-      "-> New file at /var/folders/my/lfvl285927q2hzk545f39sy40000gn/T/toree_add_deps7614799531666607290/https/repo1.maven.org/maven2/com/github/haifengl/smile-plot/1.5.1/smile-plot-1.5.1.jar\n",
+      "-> New file at /var/folders/my/lfvl285927q2hzk545f39sy40000gn/T/toree_add_deps5098665649624919110/https/repo1.maven.org/maven2/com/github/haifengl/smile-plot/1.5.1/smile-plot-1.5.1.jar\n",
       "Marking com.github.haifengl:smile-math:1.5.1 for download\n",
       "Preparing to fetch from:\n",
-      "-> file:/var/folders/my/lfvl285927q2hzk545f39sy40000gn/T/toree_add_deps7614799531666607290/\n",
+      "-> file:/var/folders/my/lfvl285927q2hzk545f39sy40000gn/T/toree_add_deps5098665649624919110/\n",
       "-> https://repo1.maven.org/maven2\n",
-      "-> New file at /var/folders/my/lfvl285927q2hzk545f39sy40000gn/T/toree_add_deps7614799531666607290/https/repo1.maven.org/maven2/com/github/haifengl/smile-math/1.5.1/smile-math-1.5.1.jar\n",
+      "-> New file at /var/folders/my/lfvl285927q2hzk545f39sy40000gn/T/toree_add_deps5098665649624919110/https/repo1.maven.org/maven2/com/github/haifengl/smile-math/1.5.1/smile-math-1.5.1.jar\n",
       "Marking com.github.haifengl:smile-core:1.5.1 for download\n",
       "Preparing to fetch from:\n",
-      "-> file:/var/folders/my/lfvl285927q2hzk545f39sy40000gn/T/toree_add_deps7614799531666607290/\n",
+      "-> file:/var/folders/my/lfvl285927q2hzk545f39sy40000gn/T/toree_add_deps5098665649624919110/\n",
       "-> https://repo1.maven.org/maven2\n",
-      "-> New file at /var/folders/my/lfvl285927q2hzk545f39sy40000gn/T/toree_add_deps7614799531666607290/https/repo1.maven.org/maven2/com/github/haifengl/smile-core/1.5.1/smile-core-1.5.1.jar\n",
+      "-> New file at /var/folders/my/lfvl285927q2hzk545f39sy40000gn/T/toree_add_deps5098665649624919110/https/repo1.maven.org/maven2/com/github/haifengl/smile-core/1.5.1/smile-core-1.5.1.jar\n",
       "Marking com.github.haifengl:smile-scala_2.11:1.5.1 for download\n",
       "Preparing to fetch from:\n",
-      "-> file:/var/folders/my/lfvl285927q2hzk545f39sy40000gn/T/toree_add_deps7614799531666607290/\n",
+      "-> file:/var/folders/my/lfvl285927q2hzk545f39sy40000gn/T/toree_add_deps5098665649624919110/\n",
       "-> https://repo1.maven.org/maven2\n",
-      "-> New file at /var/folders/my/lfvl285927q2hzk545f39sy40000gn/T/toree_add_deps7614799531666607290/https/repo1.maven.org/maven2/com/github/haifengl/smile-scala_2.11/1.5.1/smile-scala_2.11-1.5.1.jar\n",
+      "-> New file at /var/folders/my/lfvl285927q2hzk545f39sy40000gn/T/toree_add_deps5098665649624919110/https/repo1.maven.org/maven2/com/github/haifengl/smile-scala_2.11/1.5.1/smile-scala_2.11-1.5.1.jar\n",
       "Marking org.swinglabs:swingx:1.6.1 for download\n",
       "Preparing to fetch from:\n",
-      "-> file:/var/folders/my/lfvl285927q2hzk545f39sy40000gn/T/toree_add_deps7614799531666607290/\n",
+      "-> file:/var/folders/my/lfvl285927q2hzk545f39sy40000gn/T/toree_add_deps5098665649624919110/\n",
       "-> https://repo1.maven.org/maven2\n",
-      "-> New file at /var/folders/my/lfvl285927q2hzk545f39sy40000gn/T/toree_add_deps7614799531666607290/https/repo1.maven.org/maven2/org/swinglabs/swingx/1.6.1/swingx-1.6.1.jar\n",
-      "Starting download from file:/Users/julien/Documents/workspace/myrepos/spark3D/target/scala-2.11/spark3d_2.11-0.1.0.jar\n",
-      "Finished download of spark3d_2.11-0.1.0.jar\n",
-      "Using cached version of jhealpix.jar\n"
+      "-> New file at /var/folders/my/lfvl285927q2hzk545f39sy40000gn/T/toree_add_deps5098665649624919110/https/repo1.maven.org/maven2/org/swinglabs/swingx/1.6.1/swingx-1.6.1.jar\n",
+      "Starting download from file:/Users/julien/Documents/workspace/myrepos/spark3D/target/scala-2.11/spark3d_2.11-0.1.5.jar\n",
+      "Finished download of spark3d_2.11-0.1.5.jar\n",
+      "Starting download from file:/Users/julien/Documents/workspace/myrepos/spark3D/lib/jhealpix.jar\n",
+      "Finished download of jhealpix.jar\n"
      ]
     }
    ],
@@ -124,8 +123,8 @@
     "// Contains extensions to the Swing GUI toolkit\n",
     "%AddDeps org.swinglabs swingx 1.6.1\n",
     "\n",
-    "// Add the spark3d JAR. To generate it, run `sbt ++2.11.8` package at the root of the package\n",
-    "%AddJar file:/Users/julien/Documents/workspace/myrepos/spark3D/target/scala-2.11/spark3d_2.11-0.1.1.jar\n",
+    "// Add the spark3d JAR. To generate it, run `sbt ++2.11.8 package` at the root of the package\n",
+    "%AddJar file:/Users/julien/Documents/workspace/myrepos/spark3D/target/scala-2.11/spark3d_2.11-0.1.5.jar\n",
     "\n",
     "// Add healpix JAR\n",
     "%AddJar file:/Users/julien/Documents/workspace/myrepos/spark3D/lib/jhealpix.jar"
@@ -148,10 +147,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
-   "metadata": {
-    "collapsed": true
-   },
+   "execution_count": 4,
+   "metadata": {},
    "outputs": [],
    "source": [
     "import com.astrolabsoftware.spark3d.spatial3DRDD._\n",
@@ -160,13 +157,13 @@
     "\n",
     "val fnA = \"../../src/test/resources/astro_obs.fits\"\n",
     "val fnB = \"../../src/test/resources/astro_obs2.fits\"\n",
-    "val hdu = 1\n",
+    "val options = Map(\"hdu\" -> \"1\")\n",
     "val columns = \"Z_COSMO,RA,DEC\"\n",
     "val spherical = true\n",
     "\n",
     "// Load the data\n",
-    "val pointRDDA = new Point3DRDDFromFITS(spark, fnA, hdu, columns, spherical)\n",
-    "val pointRDDB = new Point3DRDDFromFITS(spark, fnB, hdu, columns, spherical)"
+    "val pointRDDA = new Point3DRDD(spark, fnA, columns, spherical, \"fits\", options)\n",
+    "val pointRDDB = new Point3DRDD(spark, fnB, columns, spherical, \"fits\", options)"
    ]
   },
   {
@@ -186,10 +183,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
-   "metadata": {
-    "collapsed": true
-   },
+   "execution_count": 5,
+   "metadata": {},
    "outputs": [],
    "source": [
     "import com.astrolabsoftware.spark3d.utils.GridType\n",
@@ -230,7 +225,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 6,
    "metadata": {},
    "outputs": [
     {
@@ -239,7 +234,7 @@
      "text": [
       "(Keeping only elements from A with counterpart in B: ,1337, points)             \n",
       "(Keeping only elements from B with counterpart in A: ,1278, points)\n",
-      "(Keeping all elements with counterparts in both A and B: ,1382, points)\n"
+      "(Keeping all elements with counterparts in both A and B: ,1337, points)         \n"
      ]
     }
    ],
@@ -284,7 +279,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 7,
    "metadata": {},
    "outputs": [
     {
@@ -323,10 +318,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
-   "metadata": {
-    "collapsed": true
-   },
+   "execution_count": 8,
+   "metadata": {},
    "outputs": [],
    "source": [
     "import smile.plot._\n",
@@ -465,10 +458,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 13,
-   "metadata": {
-    "collapsed": true
-   },
+   "execution_count": 9,
+   "metadata": {},
    "outputs": [],
    "source": [
     "val display = \"show\"\n",