shell - Getting junk character if file encoding is utf-16le

Question

For csv file which encoding is utf-16le. When I try to read data of csv it gives me junk character

To get file encoding I use below command

 file -bi test.csv

it gives me text/plain; charset=utf-16le

To read file data I use below command

head -n1 test.csv | tr '^' ','

it gives me ??colon1,colon2,colon3

Why it is giving me junk charchater

Answer 1

As the csv file is encoded with UTF-16LE, the file starts with the BOM (Byte Order Mark), 0xff and 0xfe. You can identify it with:

head -n1 test.csv | xxd

UTF-8 is most commonly used now and UTF-16 is getting less used (including Windows). Your locale will be also defaulted to UTF-8. So please try:

iconv -f UTF-16LE -t UTF-8 test.csv | head -n1 | tr '^' ','

which converts the csv file to UTF-8 coding.