Question from novice (node.js related)

[lukasMega]

Hello,
I apologize for the seemingly simple question, but I would appreciate a deeper comprehension of certain concepts of this project.

Can I use simple code/script in JS (there are no external dependencies, only few core Node.js APIs are used: net for listening on TCP port and processing binary data using Buffer API and writing to file using fs.writeFile) ?

Is it possible to run this code with txiki.js or does it require any modifications?

• probably not

If i understand correctly, i need to use (in case of writing files) https://bettercallsaghul.com/txiki.js/api/interfaces/global.tjs.FileHandle.html#write global tjs object --> FileHandle --> write method as replacement of Node.js fs module ?

Thank you for your advices.

What I want to achieve:

• create a single executable file (because it's easy to use by non-developers) and I would like to have such executable file not too big (comparing to output from Bun it has ~100 MB - .exe under Windows)
• and somehow "hide" the source code, I just need to provide an executable solution
• my concrete use-case is simple command-line application which connects (TCP) to provided IP and port where it will read binary stream of binary data and will parse that data and will produce output to file (JSON or CSV or TXT)

[saghul]

I apologize for the seemingly simple question, but I would appreciate a deeper comprehension of certain concepts of this project.

No need to apologize! Welcome!

Can I use simple code/script in JS (there are no external dependencies, only few core Node.js APIs are used: net for listening on TCP port and processing binary data using Buffer API and writing to file using fs.writeFile) ?

Is it possible to run this code with txiki.js or does it require any modifications?

* probably not

You cannot use it unmodified. This project does not implement the node API, you need to use our API for file i/o, for example.

If i understand correctly, i need to use (in case of writing files) https://bettercallsaghul.com/txiki.js/api/interfaces/global.tjs.FileHandle.html#write global tjs object --> FileHandle --> write method as replacement of Node.js fs module ?

Thank you for your advices.

Correct! You can look at the tests folder for some inspiration.

• create a single executable file (because it's easy to use by non-developers) and I would like to have such executable file not too big (comparing to output from Bun it has ~100 MB - .exe under Windows)

Yes, this will be much smaller. Note however that the standalone executable on Windows still depends on some DLLs you'll need to ship alongside your executable. I am hoping to solve that in the future.

• and somehow "hide" the source code, I just need to provide an executable solution

If you use the new tjs compile option to create your executable then the source code is hidden, as the generated bytecode is stripped from its source.

• my concrete use-case is simple command-line application which connects (TCP) to provided IP and port where it will read binary stream of binary data and will parse that data and will produce output to file (JSON or CSV or TXT)

That should be doable with the core APIs, without any external dependencies.

Cheers!

[lukasMega]

Thank you @saghul for quick reply!
I really appreciate it.

I already have a working solution written in JS (working under node.js and Bun, using pure Node.js API) and "compiled" with Bun to single executable.
Oh, actually I'm using one external npm lib for generating/validating TOTP codes - otpauth (https://github.com/hectorm/otpauth).

At first, I plan to temporarily remove this 3rd party library (otpauth) + related code and I will try to rewrite it to be txiki.js compatibile.

I will share my results later (probably tomorrow).

[lukasMega]

I'm getting

 ReferenceError: Buffer is not defined
    at <anonymous> (parserTx.js:18:11)
    at evalBytecode (native)
    at <anonymous> (run-main.js:27:495)

how to replace node.js Buffer in txiki.js ?

my source contains:

Buffer.from([2, 100, 79, 128, 101, 245, 85, 95, 95, 3, 97]),

[saghul]

You can just use an Uint8Array. Passing the array in the constructor should work.

[lukasMega]

I'm giving up.
I had hard time to rewrite node.js const readline = require('node:readline'); and it's usage to be working on txiki.js. Finally, I had success here.

But second basic thing was harder.
I'm stuck on missing some Buffer methods (writeUInt32BE) and crypto module where I'm unable to find working polyfill in order to replace missing native node core API crypto.createHmac, crypto.randomBytes, and crypto.randomInt.
I need simple TOTP functionality for validation of one-time-passwords.

Sadly, I'm leaving txiki and I'm going back to node.js/Bun.

• I'm leaving txiki for now, for this concrete project. Probably my use-case is not a good fit for txiki.js

Despite everything, I think txiki.js is great project and should deserve bigger attention.

[saghul]

The buffer methods can be replaced with DataView methods, which is the web standard way these days.

The readline module is quite complex so yeah that can be a pain.

More importantly crypto. There is no replacement in txiki.js yet :-/

I do hope to have it some day.

Thanks for giving it a try!

[lukasMega]

Unexpected update:
I'm back in game!

I solved TOTP functionality (generating and validating time-based passwords). :)

My small project overview:

   ┌─────────────┐                                
   │             │                                
   │   DEVICE    │                                
   │             │                                
   └──────┬──────┘                                
          │ RS-485                                
          │                                       
   ┌──────▼───────┐                               
   │              │     converts data from RS-485 
   │ Serial Device│◄──── and makes them available 
   │    Server    │     as TCP on specific IP:port
   └──────┬───────┘                               
          │TCP/IP                                 
          │                                       
 ┌────────▼─────────────┐   ◄─ user executes app with config file                      
 │txiki.js│             │                         
 │        │             │                         
 │  ┌─────▼──────────┐  │                         
 │  │check TOTP code │  │                         
 │  └─────┬──────────┘  │                         
 │        │             │                         
 │   ┌────▼──────────┐  │                         
 │   │ Custom parser │  │                         
 │   └────┬──────────┘  │                         
 │        │             │                         
 └────────┼─────────────┘                         
          │                                       
          ▼                                       
        ┌────┐                                    
        │file│                                    
        └────┘                                    

Right now, I need to figure out how to rewrite client logic which connects to IP:port and reads stream or Buffer of binary data to be able to run in txiki.js.
Thanks to the examples available here https://github.com/saghul/txiki.js/tree/master/examples 💪

TODO list:

• read user input (stdin) in console/terminal
• TOTP functionality
• client ... update: work in progress, handling errors ... update: done
• parser is already working in txiki.js
• generate file(s)
• read config file (.env file or json)

❓ Question
Regarding configuration possibilities in my app, I have currently .env file which is automatically loaded and available under process.env.MY_CONF_OPTION but only in Bun, not in node.js. I wanted something simple and quick with no 3rd party lib or with minimum code and Bun provides it out of the box with no code, which is nice benefit.

I'm thinking which will be simple and working solution in txiki.

• Maybe basic filesystem access and read file (json)
• or import * as config from './config.json' will work?

Do you have maybe some advice here?
Thank you

[saghul]

Importing a JSON does work out of the box!

[lukasMega]

with esbuild build step, it reads JSON in build time and inlines it's content into output bundle. I need to figure out how to prevent it and allow to import json in runtime.

[KaruroChori]

https://esbuild.github.io/api/#external
You can declare files, or patterns to be kept external.

[saghul]

Reading .env is interesting!

[lukasMega]

Is there API like in node.js to call a function before process exits ?
https://nodejs.org/api/process.html#event-beforeexit

process.on('beforeExit', (code) => {
  console.log('Process beforeExit event with code: ', code);
});

process.on('exit', (code) => {
  console.log('Process exit event with code: ', code);
});

process.on('SIGINT', handle);
process.on('SIGTERM', handle);
//...

[KaruroChori]

tjs.addSignalHandler if I recall correctly is sort of equivalent to on, or something like that.

[saghul]

Signals can be handled like @KaruroChori said and there is currently no before exit handler, though it would make sense to add a beforeunload event!

[lukasMega]

Also what I noticed quite significant memory consumption of (mostly) the same application running as compiled executable:

• using pkg tool with command: pkg -t node16-win-x64 src/server.js -o pkgServer16f.exe

  • memory size when running is 14~16 MB
  • see:

• using txiki.js compile with command: txiki.js\\build\\tjs.exe compile txbundle2.js txbundle2.exe

  • memory size when running is ~69 MB
  • see:

[saghul]

Oh I had missed this message. Interesting! Would you be able to share that file?

[lukasMega]

@saghul have you considered to make automated way (if it's possible) to install txiki.js to be more user-friendly? I mean just to run npm install -g @txikijs (to install globally) and then have available tjs executable globally ?

[saghul]

Yes, but we're not there yet because libcurl is dynamically linked so that won't work on all platforms.

I hope to address that and make a fully static binary, then this would be possible indeed.

[saghul]

I was just thinking about this now and I think it would be doable for macOS and Windows, since there we can assume (for macOS) and provide (for Windows) the required libraries.

I'll take a look when I get some cycles.