- DONE Set up ECR repos for both indexation images
- Set up indexation task (remeber to include the EFS mount)
- Test Travis build
- Test indexation
- Readme for linkliveslib (especially concerning pushing packages to nuget.org (right now done by hand))
- Check DataSync job and describe it somewhere
Dotnet application used to index person appearances, links and life courses from link-lives data (procedured by WP3) displayed on the website https://link-lives.dk (made by WP4).
The solution consists of a total of four projects: Two indexers, an indexer library and an indexer test project.
Indexers must be built individually. The project depends on LinkLivesLib which must be added as a local repository nuget package in order to be received when building the projects.
From the solution dir run: docker build -f linklives-lifecourse-indexer/Dockerfile.prod -t linklives-lifecourse-indexer .
From the solution dir run: docker build -f linklives-transcribed-indexer/Dockerfile.prod -t linklives-transcribed-indexer .
Both indexers depends on the LinkLivesLib package.
It is currently build in hand and published to nuget.org manually.
AWS Batch is used a basis for execution of the indexation. Two tasks are setup in AWS Batch: linklives-lifecourse-indexer and linklives-transcribed-indexer. Each runs an individual indexation.
Each task has two variations: Full and limited-dev. Limited-dev is used for small scale tests.
When changes are made to the source code, a new Docker image must be uploaded to AWS ECR in order to run the newest code.
All this is done using Travis-CI.
docker run linklives-lifecourse-indexer --es-host https://data-dev.link-lives.dk --path /app/link-lives/LL_data_v1.0.dev0/development --db-conn server=localhost;uid=root;pwd=123456;database=linklives_data --max-entries 10
Follow these steps to run an indexation of the test data from linklives-lib:
- Clone https://github.com/CopenhagenCityArchives/linklives-elasticsearch.git
- Start elastic search:
docker-compose -f docker-compose-dev.yml up -d - Clone https://github.com/CopenhagenCityArchives/linklives-indexer-new.git
- Run lifecourse indexer from your local machine using these parameters:
--es-host http://localhost:90 --ll-path "PATH_TO_LINKLIVESLIB/link-lives-data-demo/LL_data_v1.0/LL_data" --trs-path "PATH_TO_LINKLIVESLIB/link-lives-data-demo/transcribed_data_v1.0" --db-conn server=localhost;uid=root;pwd=123456;database=linklives_data --data-version "1.2" --skip-es false --skip-db true --max-entries 100 - Run transcribed indexer from your local machine using these parameters:
--es-host http://localhost:90 --path PATH_TO_LINKLIVESLIB\link-lives-data-demo\\transcribed_data_v1.0
The WP3 data is currently delivered by hand doing like this:
- Upload the data to S3://linklives-data
- Start a DataSync job. This will copy data from the linklives-data bucket to the EFS system used by AWS Batch