Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement FKSearch #131

Open
wants to merge 2 commits into
base: jharkhand
Choose a base branch
from
Open

Implement FKSearch #131

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b58b2df
Implement FKSearch
shashwatm1111 Jun 12, 2023
fa90332
Update
shashwatm1111 Jul 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion impl/c-qube/ingest/config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
{
"globals": {
"onlyCreateWhitelisted": true
"onlyCreateWhitelisted": true,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't let the onlyCreateWhiteList changes come into this PR.

"caseSensitiveFKSearch": false
},
"dimensions": {
"namespace": "dimensions",
Expand Down
1 change: 1 addition & 0 deletions impl/c-qube/src/services/csv-adapter/parser/utils/csvcleaner.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ describe('remove empty lines', () => {
expect(files).toEqual([
'test/fixtures/test-csvs/csvcleaner/withEmpty.csv',
'test/fixtures/test-csvs/csvcleaner/withoutEmpty.csv',
"test/fixtures/test-csvs/csvreader/fksearch.csv",
'test/fixtures/test-csvs/csvreader/invalid.reader.csv',
'test/fixtures/test-csvs/csvreader/valid.reader.csv',
'test/fixtures/test-csvs/event-grammars/test-dimension.grammar.csv',
Expand Down
22 changes: 20 additions & 2 deletions impl/c-qube/src/services/csv-adapter/parser/utils/csvreader.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { readCSV, readCSVFile } from './csvreader';
import { FKvalue, readCSV, readCSVFile } from './csvreader';

describe('CSVReader', () => {
test('parse the file fine', async () => {
Expand Down Expand Up @@ -37,4 +37,22 @@ describe('CSVReader', () => {
test('benchmarking', () => {
// add a benchmark test here similar to date parser
});
});

it('should return the correct value from the mocked config file', () => {
const configPath = 'ingest/config.json';
const result = FKvalue(configPath);
expect(result).toEqual(false);
});

it('should correctly process CSV data with case insensitivity', async () => {
const filePath = './test/fixtures/test-csvs/csvreader/fksearch.csv';
const expectedRows = [
['1', 'john', '30'],
['2', 'david', '25'],
['3', 'michael', '22'],
['4', 'mary', '28'],
];
const rows = await readCSV(filePath);
expect(rows).toEqual(expectedRows);
});
});
13 changes: 12 additions & 1 deletion impl/c-qube/src/services/csv-adapter/parser/utils/csvreader.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ const fs = require('fs').promises;

import * as csv from 'csv-parser';

const configPath = 'ingest/config.json';
export async function readCSV(filePath: string): Promise<string[][]> {
return new Promise((resolve, reject) => {
const rows: string[][] = [];
Expand All @@ -11,7 +12,11 @@ export async function readCSV(filePath: string): Promise<string[][]> {
.createReadStream(filePath)
.pipe(csv({ separator: ',', headers: false, quote: "'" }))
.on('data', (data) => {
rows.push(Object.values(data));
const rowValues: string[] = Object.values(data);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this doing?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how is this logic ensuring that case insesitivity is achieved? What if the dimensions table has a value "CQUBE" and caseSensitiveFKSearch flag is set to false and then the CSV has cqube, ideally as per the issue description this case should pass whereas in my understanding with the implementation in this PR, it will fail. Am I missing something here?

const processedRowValues = FKvalue(configPath)
? rowValues
: rowValues.map((value) => value.toLowerCase());
rows.push(processedRowValues);
})
.on('end', () => {
resolve(rows);
Expand All @@ -30,3 +35,9 @@ export async function readCSVFile(filePath: string): Promise<string[]> {
.map((row: string) => row.trim())
.filter((row: string) => row !== '');
}

export function FKvalue(configPath: string): boolean {
const configContent = fs1.readFileSync(configPath, 'utf-8');
const config = JSON.parse(configContent);
return config.globals.caseSensitiveFKSearch || false ;
}
4 changes: 4 additions & 0 deletions impl/c-qube/test/fixtures/test-csvs/csvreader/fksearch.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
1,John,30
2,David,25
3,Michael,22
4,mary,28
Loading