Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ Switch to nodenext resolution #1026

Merged
merged 7 commits into from
Nov 14, 2024
Merged

✨ Switch to nodenext resolution #1026

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
5c89e98
✨ Switch to nodenext resolution
coyotte508 Nov 14, 2024
a282b6e
✨ Fix types in published tasks package
coyotte508 Nov 14, 2024
451a223
✨ Also export snippet types
coyotte508 Nov 14, 2024
d0f9fde
⚗️ Merge remote-tracking branch 'origin/fix-tasks-types' into extensi…
coyotte508 Nov 14, 2024
3ed1667
✅ Larger timeout for gguf
coyotte508 Nov 14, 2024
31c8f4f
💡
coyotte508 Nov 14, 2024
4ea6d32
Merge remote-tracking branch 'origin/main' into extension-fix
coyotte508 Nov 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/gguf/src/gguf.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ describe("gguf", () => {
const arrayBuf = await res.arrayBuffer();
fs.writeFileSync(".cache/model.gguf", Buffer.from(arrayBuf));
}
});
}, 30_000);

it("should parse a llama2 7b", async () => {
const { metadata, tensorInfos } = await gguf(URL_LLAMA);
Expand Down
6 changes: 3 additions & 3 deletions packages/tasks/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,9 @@
},
"main": "./dist/index.cjs",
"module": "./dist/index.js",
"types": "./dist/src/index.d.ts",
"types": "./dist/index.d.ts",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious, do we need this at all? (given it's resolved automatically based on the require/import values anyway, iiuc)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I also see the second option at https://github.com/arethetypeswrong/arethetypeswrong.github.io/blob/main/docs/problems/FalseESM.md#common-causes doesn't have a types entry at all)

Copy link
Member Author

@coyotte508 coyotte508 Nov 14, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know - maybe for non-node / older contexts?

"exports": {
".": {
"types": "./dist/src/index.d.ts",
"require": "./dist/index.cjs",
"import": "./dist/index.js"
}
Expand All @@ -24,13 +23,14 @@
"format": "prettier --write .",
"format:check": "prettier --check .",
"prepublishOnly": "pnpm run inference-codegen && git diff --name-only --exit-code src && pnpm run build",
"build": "tsup src/index.ts --format cjs,esm --clean && tsc --emitDeclarationOnly --declaration",
"build": "tsup src/index.ts --format cjs,esm --clean && tsc --emitDeclarationOnly --declaration && pnpm run generate-cts",
"watch:export": "tsup src/index.ts --format cjs,esm --watch",
"watch:types": "tsc --emitDeclarationOnly --declaration --watch",
"watch": "npm-run-all --parallel watch:export watch:types",
"prepare": "pnpm run build",
"check": "tsc",
"test": "vitest run",
"generate-cts": "tsx scripts/generate-cts.ts",
"inference-codegen": "tsx scripts/inference-codegen.ts && prettier --write src/tasks/*/inference.ts",
"inference-tgi-import": "tsx scripts/inference-tgi-import.ts && prettier --write src/tasks/text-generation/spec/*.json && prettier --write src/tasks/chat-completion/spec/*.json",
"inference-tei-import": "tsx scripts/inference-tei-import.ts && prettier --write src/tasks/feature-extraction/spec/*.json"
Expand Down
19 changes: 19 additions & 0 deletions packages/tasks/scripts/generate-cts.ts
View file
Open in desktop
Copy link
Member Author

@coyotte508 coyotte508 Nov 14, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was the only way I found to have at the same time:

  • .d.cts and .d.ts generation
  • "go to source" still working

It's also possible to upgrade tsup and use --dts (instead of tsc) to generate the declaration files, but then the "go to source" won't work.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wow that feels like kind of a dubious method tbh ^^'

iiuc from https://www.typescriptlang.org/docs/handbook/modules/reference.html#node16-nodenext, the most correct way to get cjs types would be to call tsc --emitDeclarationOnly --declaration --outDir xxx after temporarily removing type: "module" from the package.json and then rename+move those declaration files; did you try that by any chance?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think you need to edit the package.json, you can run tsc with --module commonjs maybe.

But anyway I think the types are the same, only the js flies would be different. And with your solution we still need to rename / edit the .ts and .map files.

Thinking about it, we should not use tsup at all for the sourcemaps to work correctly. and in that case we do indeed need to run tsc twice. 🤔

But it was already broken, problaby

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @andrewbranch too on this btw, in case he has some insights for us by any chance on the best way to emit both ESM and CJS types through tsc directly (while transpiling the js itself through tsup) 🙏

Copy link
Member

@Pierrci Pierrci Nov 14, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think you need to edit the package.json, you can run tsc with --module commonjs maybe.

well, the thing is, it's not the same (from the link shared in my previous message):

CommonJS emit is similar to --module commonjs, but dynamic import() calls are not transformed.

^^ I guess this doesn't affect declaration files though, so maybe

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't have dynamic import calls anyway in @huggingface/tasks 🤓

(to keep in mind for @huggingface/hub though)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/isaacs/tshy automates this in a safe way

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tysm for the pointer, we'll check it out!

Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
// Just copy over the already generated .d.ts and .d.ts.map files to .d.cts and .d.cts.map files
import { readdirSync, readFileSync, statSync, writeFileSync } from "node:fs";
import { join } from "node:path";

function recursiveCopy(path: string) {
for (const item of readdirSync(path)) {
if (item.endsWith(".d.ts")) {
const content = readFileSync(join(path, item), "utf-8");
writeFileSync(join(path, item.replace(".d.ts", ".d.cts")), content.replaceAll(".d.ts.map", ".d.cts.map"));
} else if (item.endsWith(".d.ts.map")) {
const content = readFileSync(join(path, item), "utf-8");
writeFileSync(join(path, item.replace(".d.ts.map", ".d.cts.map")), content.replaceAll(".d.ts", ".d.cts"));
} else if (statSync(join(path, item)).isDirectory()) {
recursiveCopy(join(path, item));
}
}
}

recursiveCopy("dist");
4 changes: 2 additions & 2 deletions packages/tasks/src/default-widget-inputs.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import type { WidgetExample } from "./widget-example";
import type { WidgetType } from "./pipelines";
import type { WidgetExample } from "./widget-example.js";
import type { WidgetType } from "./pipelines.js";

type LanguageCode = string;

Expand Down
42 changes: 23 additions & 19 deletions packages/tasks/src/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
export { LIBRARY_TASK_MAPPING } from "./library-to-tasks";
export { MAPPING_DEFAULT_WIDGET } from "./default-widget-inputs";
export type { TaskData, TaskDemo, TaskDemoEntry, ExampleRepo } from "./tasks";
export * from "./tasks";
export { LIBRARY_TASK_MAPPING } from "./library-to-tasks.js";
export { MAPPING_DEFAULT_WIDGET } from "./default-widget-inputs.js";
export type { TaskData, TaskDemo, TaskDemoEntry, ExampleRepo } from "./tasks/index.js";
export * from "./tasks/index.js";
export {
PIPELINE_DATA,
PIPELINE_TYPES,
Expand All @@ -13,11 +13,15 @@ export {
MODALITY_LABELS,
SUBTASK_TYPES,
PIPELINE_TYPES_SET,
} from "./pipelines";
export { ALL_DISPLAY_MODEL_LIBRARY_KEYS, ALL_MODEL_LIBRARY_KEYS, MODEL_LIBRARIES_UI_ELEMENTS } from "./model-libraries";
export type { LibraryUiElement, ModelLibraryKey } from "./model-libraries";
export type { ModelData, TransformersInfo } from "./model-data";
export type { AddedToken, SpecialTokensMap, TokenizerConfig } from "./tokenizer-data";
} from "./pipelines.js";
export {
ALL_DISPLAY_MODEL_LIBRARY_KEYS,
ALL_MODEL_LIBRARY_KEYS,
MODEL_LIBRARIES_UI_ELEMENTS,
} from "./model-libraries.js";
export type { LibraryUiElement, ModelLibraryKey } from "./model-libraries.js";
export type { ModelData, TransformersInfo } from "./model-data.js";
export type { AddedToken, SpecialTokensMap, TokenizerConfig } from "./tokenizer-data.js";
export type {
WidgetExample,
WidgetExampleAttribute,
Expand All @@ -38,18 +42,18 @@ export type {
WidgetExampleOutputLabels,
WidgetExampleOutputAnswerScore,
WidgetExampleOutputText,
} from "./widget-example";
export { SPECIAL_TOKENS_ATTRIBUTES } from "./tokenizer-data";
} from "./widget-example.js";
export { SPECIAL_TOKENS_ATTRIBUTES } from "./tokenizer-data.js";

import * as snippets from "./snippets";
export * from "./gguf";
import * as snippets from "./snippets/index.js";
export * from "./gguf.js";

export { snippets };

export { SKUS, DEFAULT_MEMORY_OPTIONS } from "./hardware";
export type { HardwareSpec, SkuType } from "./hardware";
export { LOCAL_APPS } from "./local-apps";
export type { LocalApp, LocalAppKey, LocalAppSnippet } from "./local-apps";
export { SKUS, DEFAULT_MEMORY_OPTIONS } from "./hardware.js";
export type { HardwareSpec, SkuType } from "./hardware.js";
export { LOCAL_APPS } from "./local-apps.js";
export type { LocalApp, LocalAppKey, LocalAppSnippet } from "./local-apps.js";

export { DATASET_LIBRARIES_UI_ELEMENTS } from "./dataset-libraries";
export type { DatasetLibraryUiElement, DatasetLibraryKey } from "./dataset-libraries";
export { DATASET_LIBRARIES_UI_ELEMENTS } from "./dataset-libraries.js";
export type { DatasetLibraryUiElement, DatasetLibraryKey } from "./dataset-libraries.js";
4 changes: 2 additions & 2 deletions packages/tasks/src/library-to-tasks.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import type { ModelLibraryKey } from "./model-libraries";
import type { PipelineType } from "./pipelines";
import type { ModelLibraryKey } from "./model-libraries.js";
import type { PipelineType } from "./pipelines.js";

/**
* Mapping from library name to its supported tasks.
Expand Down
6 changes: 3 additions & 3 deletions packages/tasks/src/local-apps.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { parseGGUFQuantLabel } from "./gguf";
import type { ModelData } from "./model-data";
import type { PipelineType } from "./pipelines";
import { parseGGUFQuantLabel } from "./gguf.js";
import type { ModelData } from "./model-data.js";
import type { PipelineType } from "./pipelines.js";

export interface LocalAppSnippet {
/**
Expand Down
6 changes: 3 additions & 3 deletions packages/tasks/src/model-data.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import type { PipelineType } from "./pipelines";
import type { WidgetExample } from "./widget-example";
import type { TokenizerConfig } from "./tokenizer-data";
import type { PipelineType } from "./pipelines.js";
import type { WidgetExample } from "./widget-example.js";
import type { TokenizerConfig } from "./tokenizer-data.js";

/**
* Public interface for model metadata
Expand Down
6 changes: 3 additions & 3 deletions packages/tasks/src/model-libraries-snippets.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import type { ModelData } from "./model-data";
import type { WidgetExampleTextInput, WidgetExampleSentenceSimilarityInput } from "./widget-example";
import { LIBRARY_TASK_MAPPING } from "./library-to-tasks";
import type { ModelData } from "./model-data.js";
import type { WidgetExampleTextInput, WidgetExampleSentenceSimilarityInput } from "./widget-example.js";
import { LIBRARY_TASK_MAPPING } from "./library-to-tasks.js";

const TAG_CUSTOM_CODE = "custom_code";

Expand Down
6 changes: 3 additions & 3 deletions packages/tasks/src/model-libraries.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import * as snippets from "./model-libraries-snippets";
import type { ModelData } from "./model-data";
import type { ElasticSearchQuery } from "./model-libraries-downloads";
import * as snippets from "./model-libraries-snippets.js";
import type { ModelData } from "./model-data.js";
import type { ElasticSearchQuery } from "./model-libraries-downloads.js";

/**
* Elements configurable by a model library.
Expand Down
2 changes: 1 addition & 1 deletion packages/tasks/src/snippets/common.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import type { ChatCompletionInputMessage, GenerationParameters } from "../tasks";
import type { ChatCompletionInputMessage, GenerationParameters } from "../tasks/index.js";

export function stringifyMessages(
messages: ChatCompletionInputMessage[],
Expand Down
4 changes: 2 additions & 2 deletions packages/tasks/src/snippets/curl.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import type { ModelDataMinimal } from "./types";
import type { ModelDataMinimal } from "./types.js";
import { describe, expect, it } from "vitest";
import { snippetTextGeneration } from "./curl";
import { snippetTextGeneration } from "./curl.js";

describe("inference API snippets", () => {
it("conversational llm", async () => {
Expand Down
9 changes: 5 additions & 4 deletions packages/tasks/src/snippets/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import * as inputs from "./inputs";
import * as curl from "./curl";
import * as python from "./python";
import * as js from "./js";
import * as inputs from "./inputs.js";
import * as curl from "./curl.js";
import * as python from "./python.js";
import * as js from "./js.js";
export * from "./types.js";

export { inputs, curl, python, js };
6 changes: 3 additions & 3 deletions packages/tasks/src/snippets/inputs.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import type { PipelineType } from "../pipelines";
import type { ChatCompletionInputMessage } from "../tasks";
import type { ModelDataMinimal } from "./types";
import type { PipelineType } from "../pipelines.js";
import type { ChatCompletionInputMessage } from "../tasks/index.js";
import type { ModelDataMinimal } from "./types.js";

const inputsZeroShotClassification = () =>
`"Hi, I recently bought a device from your company but it is not working as advertised and I would like to get reimbursed!"`;
Expand Down
4 changes: 2 additions & 2 deletions packages/tasks/src/snippets/js.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import type { InferenceSnippet, ModelDataMinimal } from "./types";
import type { InferenceSnippet, ModelDataMinimal } from "./types.js";
import { describe, expect, it } from "vitest";
import { snippetTextGeneration } from "./js";
import { snippetTextGeneration } from "./js.js";

describe("inference API snippets", () => {
it("conversational llm", async () => {
Expand Down
4 changes: 2 additions & 2 deletions packages/tasks/src/snippets/python.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import type { ModelDataMinimal } from "./types";
import type { ModelDataMinimal } from "./types.js";
import { describe, expect, it } from "vitest";
import { snippetConversational } from "./python";
import { snippetConversational } from "./python.js";

describe("inference API snippets", () => {
it("conversational llm", async () => {
Expand Down
2 changes: 1 addition & 1 deletion packages/tasks/src/snippets/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import type { ModelData } from "../model-data";
import type { ModelData } from "../model-data.js";

/**
* Minimal model data required for snippets.
Expand Down
2 changes: 1 addition & 1 deletion packages/tasks/src/tasks/audio-classification/data.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import type { TaskDataCustom } from "..";
import type { TaskDataCustom } from "../index.js";

const taskData: TaskDataCustom = {
datasets: [
Expand Down
2 changes: 1 addition & 1 deletion packages/tasks/src/tasks/audio-to-audio/data.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import type { TaskDataCustom } from "..";
import type { TaskDataCustom } from "../index.js";

const taskData: TaskDataCustom = {
datasets: [
Expand Down
2 changes: 1 addition & 1 deletion packages/tasks/src/tasks/automatic-speech-recognition/data.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import type { TaskDataCustom } from "..";
import type { TaskDataCustom } from "../index.js";

const taskData: TaskDataCustom = {
datasets: [
Expand Down
2 changes: 1 addition & 1 deletion packages/tasks/src/tasks/depth-estimation/data.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import type { TaskDataCustom } from "..";
import type { TaskDataCustom } from "../index.js";

const taskData: TaskDataCustom = {
datasets: [
Expand Down
2 changes: 1 addition & 1 deletion packages/tasks/src/tasks/document-question-answering/data.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import type { TaskDataCustom } from "..";
import type { TaskDataCustom } from "../index.js";

const taskData: TaskDataCustom = {
datasets: [
Expand Down
2 changes: 1 addition & 1 deletion packages/tasks/src/tasks/feature-extraction/data.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import type { TaskDataCustom } from "..";
import type { TaskDataCustom } from "../index.js";

const taskData: TaskDataCustom = {
datasets: [
Expand Down
2 changes: 1 addition & 1 deletion packages/tasks/src/tasks/fill-mask/data.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import type { TaskDataCustom } from "..";
import type { TaskDataCustom } from "../index.js";

const taskData: TaskDataCustom = {
datasets: [
Expand Down
2 changes: 1 addition & 1 deletion packages/tasks/src/tasks/image-classification/data.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import type { TaskDataCustom } from "..";
import type { TaskDataCustom } from "../index.js";

const taskData: TaskDataCustom = {
datasets: [
Expand Down
2 changes: 1 addition & 1 deletion packages/tasks/src/tasks/image-feature-extraction/data.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import type { TaskDataCustom } from "..";
import type { TaskDataCustom } from "../index.js";

const taskData: TaskDataCustom = {
datasets: [
Expand Down
2 changes: 1 addition & 1 deletion packages/tasks/src/tasks/image-segmentation/data.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import type { TaskDataCustom } from "..";
import type { TaskDataCustom } from "../index.js";

const taskData: TaskDataCustom = {
datasets: [
Expand Down
2 changes: 1 addition & 1 deletion packages/tasks/src/tasks/image-text-to-text/data.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import type { TaskDataCustom } from "..";
import type { TaskDataCustom } from "../index.js";

const taskData: TaskDataCustom = {
datasets: [
Expand Down
2 changes: 1 addition & 1 deletion packages/tasks/src/tasks/image-to-3d/data.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import type { TaskDataCustom } from "..";
import type { TaskDataCustom } from "../index.js";

const taskData: TaskDataCustom = {
datasets: [
Expand Down
2 changes: 1 addition & 1 deletion packages/tasks/src/tasks/image-to-image/data.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import type { TaskDataCustom } from "..";
import type { TaskDataCustom } from "../index.js";

const taskData: TaskDataCustom = {
datasets: [
Expand Down
2 changes: 1 addition & 1 deletion packages/tasks/src/tasks/image-to-text/data.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import type { TaskDataCustom } from "..";
import type { TaskDataCustom } from "../index.js";

const taskData: TaskDataCustom = {
datasets: [
Expand Down
Loading
Loading