const char *text : string or unistring

[nico-robert]

Hello @apnadkarni ,

Always working on my library (libharu).
C function HPDF_Page_TextOut :

HPDF_Page_TextOut  (HPDF_Page    page,
                    HPDF_REAL    xpos,
                    HPDF_REAL    ypos,
                    const char  *text);

cffi function :

    HPDF_Page_TextOut HPDF_STATUS {
        page HPDF_Page
        xpos HPDF_REAL
        ypos HPDF_REAL
        text string
    }

text define to string works in most cases but when I have symbol in my string the display of characters is strange.
If I set the text to unistring it works with different encodings and different symbols, but it only displays the first character, by example :
HPDF_Page_TextOut $page 10 10 "Text_example ©"
display : T
If I set the text to string , Text_example works but for copyright symbol my display is like this :
Text_example ́'
I'm a little confused... What type I should put as argument according to my text ?

I hope I was clear in my request

Thanks
Nicolas

[apnadkarni]

@nico-robert Glad you are plugging away :-)

Regarding the encoding with haru, see this.

Basically you need to tell haru what text encoding to use by calling HPDF_SetCurrentEncoder and setting it to utf8 as in that link. Then in the CFFI definition use parameter type string.utf-8 to tell CFFI to pass strings to that function in utf8 encoding.

Let me know if that works, or doesn't.

/Ashok

[nico-robert]

yes I keep trying... I am trying to repeat the examples written in C (demo)
For the moment I am at this file : slide_show_demo.c
I tried following the link you gave me ,I already used it to load a *ttf file and the method works, but in my case same result...
I thought it worked using the unistring type because I used it for this file encoding_list.c it was probably beginner's luck

PS : A few days ago I realized that another person had already written this in Tcl (tclhpdf) 😥

[apnadkarni]

I wasn't aware of the tclhpdf either. That's always a danger with open source. The release at that link seems quite old though, so there may still be value to a newer implementation.

I suspect your unicode worked with encoding_list.c because it seems to write only one char at a time in draw_text and only letters between 0x20 and 0xff in which both unicode and almost any other encoding will look the same (the single byte followed by a zero byte).

Regarding your current problem, can you make your code available? I can try taking a look at the problem but I don't have time to write all the cffi interface myself to test it out.

/Ashok

[nico-robert]

Hello @apnadkarni ,
How can I share it with you ?

[apnadkarni]

You could just attach it to your reply by clicking at the bottom of the github discussions entry widget (see where it says "Attach files..." unless you do not want to make it public.

[nico-robert]

Hello @apnadkarni ,
Sorry for the delay...

Please find attached Haru project

2 examples in the demo folder following our conversation :

• slide_show_demo.tcl
• encoding_list.tcl

Happy new year
Nicolas

[apnadkarni]

Thanks, I'll take a look in the next couple of days. Just need to finish up some enhancements to cffi first.

/Ashok

[apnadkarni]

I had a quick look. Attached is a patch which works for this demo (only looked at encoding_list). Although only three lines have changed, the ramifications are wide and deep in terms of how the interface should look. I had misunderstood the hpdf api so need to think about it a bit more.

BTW, I was impressed with how far along you are with the interface and demos.

/Ashok
haru.patch.txt

[nico-robert]

Hello @apnadkarni ,
You changed this in the code (string by binary):

    HPDF_Page_TextOut HPDF_STATUS {
        page HPDF_Page
        xpos HPDF_REAL
        ypos HPDF_REAL
        text binary
    }

Because If the type is string I have the same result

Thanks
Nicolas

[apnadkarni]

Yes, that change is necessary. The short answer is that the string type is used when the called function always uses a single encoding. So for example if it always expects the system encoding the type can be string, if it always expects UTF8, the parameter type can be string.utf-8 and so on. In such cases, you can directly pass Tcl string to the function and have the encodings automatically done for you.

In the case of HPDF_Page_TextOut and HPDF_Page_MeasureText (and I would guess the hpdf text functions in general), they do not take the same encoding every time. One way to handle this is to do the encoding transformation explicitly in Tcl using the encoding convertto function and use binary as the parameter type so CFFI itself will not do any implicit transforms.

Also, I assume you understood the reason for the following change:

-                set buffer [format %c $buf]
+                set buffer [binary format cc $buf 0]

The first line sets buffer to a Unicode character corresponding to a specific byte value, say 0xA3 (for example). However, the intent is not to interpret the 0xA3 as a Unicode char but rather 0xA3 as an encoding using the encoding being displayed e.g. iso8859-2. That is what the second line does, it stores the byte value 0xa3 followed by the terminating \0 character.

Now the above works for when you want to write a single character corresponding to an specific ordinal position in the encoding. For outputting a general Unicode string, the application would need to do something like

set text "Some string"
set encoded_text [encoding convertto iso8859-2 $text]
append encoded_text \0; # Or \0\0 if double byte encodings
HPDF_Page_TextOut $page $xpos $ypos $encoded_text

Hope that clarifies but I'll elaborate later when I've more time.

/Ashok

[nico-robert]

Thanks for the clarification...
A idea... cffi would be able to detect if text value is a byte value ?
If the value is proceeded as byte value, no transformation otherwise the value is proceeded as string.utf-8 and convert to external encoding.
However, the question I ask myself following my suggestion , is how you can know that it is the current encoding ?

[apnadkarni]

To conclude my previous post, mu suggestion is that all parameters that are expected to be encoded strings be changed to binary type. In addition, when passing string arguments for these parameters, applications use the hpdf_encode proc as below (not tested).

proc hpdf_encode {text hpdf_encoding} {
    set tcl_encodings {
        "StandardEncoding" cp1252
        "MacRomanEncoding" macroman
        "WinAnsiEncoding" cp1252
        "ISO8859-2" iso8859-2
        "ISO8859-3" iso8859-3
        "ISO8859-4" iso8859-4
        "ISO8859-5" iso8859-5
        "ISO8859-9" iso8859-9
        "ISO8859-10" iso8859-10
        "ISO8859-13" iso8859-13
        "ISO8859-14" iso8859-14
        "ISO8859-15" iso8859-15
        "ISO8859-16" iso8859-16
        "CP1250" cp1250
        "CP1251" cp1251
        "CP1252" cp1252
        "CP1254" cp1254
        "CP1257" cp1257
        "KOI8-R" koi8-r
        "Symbol-Set" symbol
        "ZapfDingbats-Set" dingbats
    }
   # Note: two nulls appended in case encoding is double-byte
    append encoded [encoding convertto [dict get $tcl_encodings $hpdf_encoding] $text] "\0\0"
    return $encoded
}

Then an example call would look like

HPDF_Page_TextOut $page $xpos $ypos [hpdf_encode "abcd" ISO8859-2]

[apnadkarni]

Also, to reply to your question, there is no way for cffi to distinguish between bytes vs strings vs any other types thanks to Tcl's everything is a string semantics.

[nico-robert]

Thanks , I don’t know if I’m able to find the encoding applied to my font. I'm gonna keep looking...

[apnadkarni]

When you say encoding for your font, do you mean the font you are using for testing? My understanding is that the application has to enable the appropriate fonts when calling haru so from the interface point of view, it does not have to detect the font. May be I misunderstand the issue.

/Ashok

[nico-robert]

Yes my encoding for my font I use... If I understand your procedure hpdf_encode I need to know enconding which is applied to my font I'm right ?
For example : set font [HPDF_GetFont $pdf "Courier" "StandardEncoding"]
In this case I need to know this : StandardEncoding
I'm confused...
I compiled slide_show_demo.c with this string with copyright symbol : "Type \"Ctrl+L\" in order to return from full screen mode ©"
and the result :

• Type "Ctrl+L" in order to return from full screen mode ́'

In Tcl I have this result with your procedure hpdf_encode and cp1252 as encoding :

• Type "Ctrl+L" in order to return from full screen mode '

Without your procedure I have this result :

• Type "Ctrl+L" in order to return from full screen mode ' 't ¤ (It's different for each slide)

In any case, this I think does not come from your library (I think you already knew 😉), it is probably me who does not understand libharu
Maybe , for this case , I have to write this function HPDF_Page_TextOut without the symbol and 2nd call function just with the symbol in this form : [binary format cc $symbol 0]

Thanks anyway
Nicolas

[apnadkarni]

Yes, my expectation is the caller needs to know the encoding.

I'm confused too about libharu usage. It is not clear to me why both the font related and text output calls require specification of the encoding. I would have thought, the encoding is set once, and then used in all calls until it is set again.

I had a quick look at the older haru Tcl binding you linked to. That also has the same issue as far as I could tell - you have to explicitly do the encoding before calling the bound functions.

BTW, how are you constructing the copyright symbol in the Tcl code? Using \u notation or something else?

[nico-robert]

Hello @apnadkarni ,
2 ways : like this © and like this \u00A9 without success
I even tried this with encoding to : "ISO8859-13" and this command [binary format cc 169 0] and the result is : Ā©

[apnadkarni]

The following worked for me (patch against your original code).

<     HPDF_Page_TextOut $page 55 300 "Type 'Ctrl+L' in order to return from full screen mode."
---
>     HPDF_Page_TextOut $page 55 300 [hpdf_encode "Type 'Ctrl+L' in order to return from full screen mode!\u00a9" CP1252]
67c96
< set font [HPDF_GetFont $pdf "Courier" ""]
---
> set font [HPDF_GetFont $pdf "Courier" "CP1252"]

Line numbers likely off as I had some "experimental" code in there as well.

The key I think is that the third argument to HPDF_GetFont should be the same as the encoding used when outputting text via HPDF_Page_TextOut (CP1252 here but likely any would do as long as it is the same in both calls)

The (c) then displays correctly, and you might notice the single quotes around Ctrl-L are also fixed.

/Ashok

[apnadkarni]

By the way, you need to also add [hpdf_encode...] when outputting the caption or any other text.

[nico-robert]

It works... I will continue in this way.
Thanks for the support
Good luck for the future... of tcl-cffi

Nicolas

[apnadkarni]

Thanks. Pls let me know if you open source your work.

/Ashok

[nico-robert]

Thanks. Pls let me know if you open source your work.
/Ashok

@apnadkarni done 😉

[apnadkarni]

That's great! Glad you persevered! By the way, I released 1.0b4 in the meanwhile. Some new conveniences but also some incompatibilities with 1.0b3 which I hope do not affect you. The API should be stable now for a 1.0 release.

Can you put up a link to your code on the wiki? Or I can if you don't log on to the wiki.

[nico-robert]

Tested this morning with v1.0b4 no problem. wiki link

Nicolas