Special/Unicode Characters Not Parsing Properly

[sagargg]

I have encountered an issue where special or Unicode characters within the values of certain data eg. € are not parsing properly.

Example of CSV:

Name,Age,City,Amount
John,25,New York,$2000
Jane,30,London, €2000

On Data Preview:

On Data API:

On qsvdp CLI:

[TomeCirun]

Working fine on the dev-v1.0 branch with QSV version 0.91.0 @sagargg

We are currently testing and preparing the DP+ for a version 1.0 release
more info here

[jqnatividad]

Hi @sagargg ,
Can you confirm what version of DP+ you're using? 0.15.0 should also work without issues.

If its still a problem, what's the encoding of the file you're uploading?

DP+ normalizes non UTF8-encoded files to their "lossy" encoding - i.e. replacing invalid UTF-8 sequences with U+FFFD REPLACEMENT CHARACTER, which is the � character you're getting instead of the euro currency symbol.

[sagargg]

@jqnatividad Maybe it is happening only on macOS.

Even it is happening on the latest version of qsv.

[jqnatividad]

@sagargg , do you mind uploading a sample file here?

I also have macOS (M2 Pro Mac Mini) and it works without issues.

testeuro.csv

$ qsvdp --version
qsvdp 0.108.0-standard-Luau 0.581;self_update-12-12;25.60 GiB-0B-0B-32.00 GiB (aarch64-apple-darwin compiled with Rust 1.70.0) prebuilt

$ qsvdp input testeuro.csv 
Name,Age,City,Amount
John,25,New York,$2000
Jane,30,London,€2000

$ qsvdp sniff testeuro.csv 
Path: /Users/joelnatividad/Local/GitHub/qsv/testeuro.csv
Sniff Timestamp: 2023-07-07T04:07:46.429694+00:00
Last Modified: 2023-07-07T04:06:33+00:00
Delimiter: ,
Header Row: true
Preamble Rows: 0
Quote Char: none
Flexible: false
Is UTF8: true
Detected Mime Type: application/csv
Retrieved Size (bytes): 67
File Size (bytes): 67
Sampled Records: 2
Estimated: false
Num Records: 2
Avg Record Len (bytes): 19
Num Fields: 4
Stats Types: false
Fields:
    0:  Text      Name
    1:  Unsigned  Age
    2:  Text      City
    3:  Text      Amount

[jqnatividad]

BTW, I just noticed you're on a very old version of qsv - 0.92.0.

You should upgrade to qsv 0.108.0

[sagargg]

@jqnatividad yeah, it worked with the file that you've shared.

Here is the file i'm using.
csv.csv

[jqnatividad]

The encoding of the file is not utf-8, that's why qsv input did a lossy-transcoding to utf8.

$ file -I csv.csv
csv.csv: text/plain; charset=iso-8859-1

$ file -I testeuro.csv 
testeuro.csv: text/csv; charset=utf-8

[sagargg]

okay. Now I understood, Thank you for clarifying.