Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mangled output windows commandline when using wstr and pipe #420

Open
thrive4 opened this issue Oct 3, 2023 · 0 comments
Open

mangled output windows commandline when using wstr and pipe #420

thrive4 opened this issue Oct 3, 2023 · 0 comments

Comments

@thrive4
Copy link

thrive4 commented Oct 3, 2023

Not sure to what degree this is os related, the commandline
aka console is using the standard US oem codepage with both
windows 7 and windows 10, however when using wstr to output
text in combination with ' | more' or ' > ' aka piping output
the output is mangled.

Tested on windows 7 and windows 10
fbc 1.09 and fbc 1.10.

Sample code to illustrate the issue:

dim help as string
help = "example: cmdsqlite.exe game.db select name from game where name like 'a%'"
select case command()
    case "-h", "-help", "--help", "-man"
        print wstr(help)
    case "-nowstr"
        print help
    case else
        print "invalid switch"
        end
end select

Compile above code (by example as wstr_issue.exe) and run:
Normal output
wstr_issue.exe -h

Mangled output
wstr_issue.exe -h | more
or
wstr_issue.exe -h > test.txt

Normal output
wstr_issue.exe -nowstr | more
wstr_issue.exe -nowstr > test.txt

In depth info commandline see:
https://devblogs.microsoft.com/commandline/windows-command-line-unicode-and-utf-8-output-text-buffer/
https://devblogs.microsoft.com/commandline/windows-command-line-introducing-the-windows-pseudo-console-conpty/

thrive4 added a commit to thrive4/util.fb.cmdsqlite that referenced this issue Mar 10, 2025
- started a baseline sqlite fts5 implementation
  import datasources csv, json, folder, xml
  and rundimentarty text and exif info specific filetypes
  issues: fts5
  'rank' is a reserved name so the fieldname needs to be renamed
  there are issues with hyphen in table names ....
- gradual implementation json data format escaping rules
  import pjson (aka pretty json), single and multi line
  with basic error handling
- started a rundementary form of disk catalog format
  drive label, folder, folder date, folder size
  example: cmdsqlite e:\ catalog
- added basic comment handling in .query comment character is # (hash)
- added a number of sample .query files
- added basic .txt import and encoding support utf16 and utf8 default 'ascii'
  list version each line of text is record
- added basic .srt import format
  file, subtitlenr, starttime, endtime, text
- added theme field to mp3 html export theme is last folder in path
  exampe: theme = 'classic' from g:\data\mp3\classic\
- added file metrics folder size, drive label, drive storage
- added support database extension .sqlite used by some applications and browsers
- removed ' in xml node names
- regression checkpath (see 1.1) fixed again
- more fixes to errorhandling csv import
- fixed incorrect table defintion xml import
- fixed preserve linefeed when converting .sql to .db
- fixed issue with numerical data in json pattern "fieldname":123
  becomes "fieldname":"123"
- issue with wstr still open freebasic/fbc#420
  temporary hack check locale if not 'en' or 'nl' use wstr
- updated sqlite3.dll v3.42.0 to v3.49.1
- updated help files and readme.md where needed
- slimmed down function explode
- tweaked getmp3tag increase speed parsing mp3 tags
- improved getfilesfromfolder and getfolders in utilfile.bas

! experimental !
notes regarding text import:

generate index per file:
cmdsqlite.exe f:\dev\freebasic\projects\cmdsqlite\data\indextest *.txt json index
generate sql with content text per file:
cmdsqlite.exe f:\dev\freebasic\projects\cmdsqlite\data\indextest *.txt sql exif
generate sql with content text per file fts:
cmdsqlite.exe f:\dev\freebasic\projects\cmdsqlite\data\indextest *.txt sql exif fts

work in progress bom (utf8 and utf16) default = no pragma
cmdsqlite>cmdsqlite.exe f:\dev\freebasic\projects\cmdsqlite\data\indextest *.txt sql exif utf8
pragma encoding = utf8;

cmdsqlite>cmdsqlite.exe f:\dev\freebasic\projects\cmdsqlite\data\indextest *.txt sql exif utf16
pragma encoding = utf8;
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant