Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Relative path definition in "paths" mapping not working properly #1755

Open
lachniej opened this issue Sep 6, 2024 · 4 comments
Open

Relative path definition in "paths" mapping not working properly #1755

lachniej opened this issue Sep 6, 2024 · 4 comments
Assignees
@pietern
Labels
Bug Something isn't working DABs DABs related issues

Comments

@lachniej
Copy link

lachniej commented Sep 6, 2024 •
edited
Loading

In attempting to use the paths mapping within the sync in a databricks.yml I am getting various errors when attempting to sync a common folder at the same level as the bundle directory (not the bundle root). It is producing errors both with pydabs and dabs default initialization templates.

Configuration

databricks cli version: 0.227.0 and 0.227.1
pydabs version for pydabs testing: 0.5.1
python version: 3.11

Folder Structure - dabs --using default dabs template--

├── common/
│   ├── src/
│   │  ├── __init__.py
│   │  ├── hello.py
└── my_bundle/
│   ├── databrick.yml
│   ├── src/
│   ├── dist/
│   │  ├── test_this-0.0.1+20240906.124245-py3-none-any.whl
│   ├── resources/
│   │  ├── my_bundle_job.yml
│   ├── ...

Folder Structure - pydabs --using default pydabs template--

├── common/
│   ├── src/
│   │  ├── __init__.py
│   │  ├── hello.py
└── my_bundle/
│   ├── databrick.yml
│   ├── src/
│   │  ├── my_bundle/ 

│   ├── ...

databricks.yml - dabs

# This is a Databricks asset bundle definition for test_this.
# See https://docs.databricks.com/dev-tools/bundles/index.html for documentation.
bundle:
  name: test_this

include:
  - resources/*.yml

sync:
  paths:
    - ../common

targets:
  # The 'dev' target, for development purposes. This target is the default.
  dev:
    # We use 'mode: development' to indicate this is a personal development copy:
    # - Deployed resources get prefixed with '[dev my_user_name]'
    # - Any job schedules and triggers are paused by default
    # - The 'development' mode is used for Delta Live Tables pipelines
    mode: development
    default: true
    workspace:
      host: https://adb-xxx-azuredatabricks.net

  ## Optionally, there could be a 'staging' target here.
  ## (See Databricks docs on CI/CD at https://docs.databricks.com/dev-tools/bundles/ci-cd.html.)
  #
  # staging:
  #   workspace:
  #     host: https://adb-xxx-azuredatabricks.net

  # The 'prod' target, used for production deployment.
  prod:
    # We use 'mode: production' to indicate this is a production deployment.
    # Doing so enables strict verification of the settings below.
    mode: production
    workspace:
      host: https://adb-xxx-azuredatabricks.net
      # We always use /Users/[email protected] for all resources to make sure we only have a single copy.
      # If this path results in an error, please make sure you have a recent version of the CLI installed.
      root_path: /Users/[email protected]/.bundle/${bundle.name}/${bundle.target}
    run_as:
      # This runs as [email protected] in production. We could also use a service principal here,
      # see https://docs.databricks.com/dev-tools/bundles/permissions.html.
      user_name: [email protected]

resources/my_bundle_job.yml - dabs

resources:
  jobs:
    my_bundle_job:
      name: my_bundle_job

      schedule:
        # Run every day at 8:37 AM
        quartz_cron_expression: '44 37 8 * * ?'
        timezone_id: Europe/Amsterdam

      email_notifications:
        on_failure:
          - [email protected]

      tasks:
        - task_key: notebook_task
          job_cluster_key: job_cluster
          notebook_task:
            notebook_path: ../src/notebook.ipynb
        
        - task_key: refresh_pipeline
          depends_on:
            - task_key: notebook_task
          pipeline_task:
            pipeline_id: ${resources.pipelines.test_this_pipeline.id}
        
        - task_key: main_task
          depends_on:
            - task_key: refresh_pipeline
          job_cluster_key: job_cluster
          python_wheel_task:
            package_name: test_this
            entry_point: main
          libraries:
            # By default we just include the .whl file generated for the test_this package.
            # See https://docs.databricks.com/dev-tools/bundles/library-dependencies.html
            # for more information on how to add other libraries.
            - whl: ../dist/*.whl

      job_clusters:
        - job_cluster_key: job_cluster
          new_cluster:
            spark_version: 13.3.x-scala2.12
            node_type_id: Standard_D3_v2
            autoscale:
                min_workers: 1
                max_workers: 4

databricks.yml - pydabs

# This is a Databricks asset bundle definition for pydabs_5_1.
# See https://docs.databricks.com/dev-tools/bundles/index.html for documentation.
bundle:
  name: pydabs_5_1
  databricks_cli_version: ">= 0.227.1"

include:
  - "resources/*.yml"

experimental:
  pydabs:
    enabled: true
    venv_path: .venv
    # list of Python packages to import to discover resources, resource generator
    # and mutators
    import:
    - "pydabs_5_1"
  scripts:
    preinit: |
      # create .venv if it doesn't exist
      if [ ! -e .venv ]; then
        python=$(command -v python3.10 || command -v python3.11 || command -v python3.12) || (
          echo "ERROR: Python 3.10 or higher is required to create virtual environment";
          exit 1;
        )

        $python -m venv .venv
        .venv/bin/pip install -r requirements-dev.txt
      fi

artifacts:
  default:
    type: whl
    path: .
    build: .venv/bin/python3 setup.py bdist_wheel

sync:
  exclude:
    - "src"
  paths:
    - ../common

variables:
  warehouse_id:
    description: Warehouse ID for SQL tasks
    default: change-me
  default_cluster_spec:
    description: Spec for 'Default' job cluster
    default:
      spark_version: 13.3.x-scala2.12
      node_type_id: Standard_D3_v2
      autoscale:
        min_workers: 1
        max_workers: 4

targets:
  # The 'dev' target, used for development purposes.
  # Whenever a developer deploys using 'dev', they get their own copy.
  dev:
    # We use 'mode: development' to make sure everything deployed to this target gets a prefix
    # like '[dev my_user_name]'. Setting this mode also disables any schedules and
    # automatic triggers for jobs and enables the 'development' mode for Delta Live Tables pipelines.
    mode: development
    default: true
    workspace:
      host: https://adb-xxx-azuredatabricks.net

  # Optionally, there could be a 'staging' target here.
  # (See Databricks docs on CI/CD at https://docs.databricks.com/dev-tools/bundles/index.html.)
  #
  # staging:
  #  workspace:
  #    host: https://adb-xxx-azuredatabricks.net

  # The 'prod' target, used for production deployment.
  prod:
    # For production deployments, we only have a single copy, so we override the
    # workspace.root_path default of
    # /Users/${workspace.current_user.userName}/.bundle/${bundle.target}/${bundle.name}
    # to a path that is not specific to the current user.
    mode: production
    workspace:
      host: https://adb-xxx-azuredatabricks.net
      root_path: /Shared/.bundle/prod/${bundle.name}
    run_as:
      # This runs as [email protected] in production. Alternatively,
      # a service principal could be used here using service_principal_name
      # (see Databricks documentation).
      user_name: [email protected]

Steps to reproduce the behavior

Please list the steps required to reproduce the issue, for example:

databricks bundle deploy --target dev

Expected Behavior

Expected behavior would be deployment of bundle along with the shared package for use across multiple bundles within the same repo

Actual Behavior

error - dabs

Error: no files match pattern: my_bundl/dist/*.whl
  at resources.jobs.my_bundle_job.tasks[2].libraries[0].whl
  in resources/my_bundle_job.yml:39:15

error - pydabs

Error: build failed default, error: chdir /Users/xxx/git_repos/my_bundle/my_bundle/my_bundle: no such file or directory

OS and CLI version

tested this with databricks cli version: 0.227.0 and 0.227.1

Is this a regression?

does not work in neither version since feature release

Debug Logs

dabs

09:43:08  INFO start pid=57009 version=0.227.0 args="databricks, bundle, deploy, --target, dev, --log-level=debug"
09:43:08 DEBUG Found bundle root at /Users/xxx/git_repos/pydabs5_1/pydabs_5_1 (file /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/databricks.yml) pid=57009
09:43:08 DEBUG Apply pid=57009 mutator=load
09:43:08  INFO Phase: load pid=57009 mutator=load
09:43:08 DEBUG Apply pid=57009 mutator=load mutator=seq
09:43:08 DEBUG Apply pid=57009 mutator=load mutator=seq mutator=EntryPoint
09:43:08 DEBUG Apply pid=57009 mutator=load mutator=seq mutator=scripts.preinit
Executing 'preinit' script
09:43:08 DEBUG Apply pid=57009 mutator=load mutator=seq mutator=ProcessRootIncludes
09:43:08 DEBUG Apply pid=57009 mutator=load mutator=seq mutator=ProcessRootIncludes mutator=seq
09:43:08 DEBUG Apply pid=57009 mutator=load mutator=seq mutator=VerifyCliVersion
09:43:08 ERROR Error: Databricks CLI version constraint not satisfied. Required: >= 0.227.1, current: 0.227.0 pid=57009 mutator=load mutator=seq mutator=VerifyCliVersion
09:43:08 ERROR Error: Databricks CLI version constraint not satisfied. Required: >= 0.227.1, current: 0.227.0 pid=57009 mutator=load mutator=seq
09:43:08 ERROR Error: Databricks CLI version constraint not satisfied. Required: >= 0.227.1, current: 0.227.0 pid=57009 mutator=load
Error: Databricks CLI version constraint not satisfied. Required: >= 0.227.1, current: 0.227.0

09:43:08 ERROR failed execution pid=57009 exit_code=1 error=AlreadyPrinted

pydabs

09:44:45  INFO start pid=57142 version=0.227.0 args="databricks, bundle, deploy, --target, dev, --log-level, debug"
09:44:45 DEBUG Found bundle root at /Users/xxx/git_repos/test_this (file /Users/xxx/git_repos/test_this/databricks.yml) pid=57142
09:44:45 DEBUG Apply pid=57142 mutator=load
09:44:45  INFO Phase: load pid=57142 mutator=load
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=EntryPoint
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=scripts.preinit
09:44:45 DEBUG No script defined for preinit, skipping pid=57142 mutator=load mutator=seq mutator=scripts.preinit
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=ProcessRootIncludes
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=ProcessRootIncludes mutator=seq
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=ProcessRootIncludes mutator=seq mutator=ProcessInclude(resources/test_this_job.yml)
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=ProcessRootIncludes mutator=seq mutator=ProcessInclude(resources/test_this_pipeline.yml)
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=VerifyCliVersion
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=EnvironmentsToTargets
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=InitializeVariables
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=DefineDefaultTarget(default)
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=LoadGitDetails
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=PythonMutator(load)
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=validate:unique_resource_keys
09:44:45 DEBUG Apply pid=57142 mutator=load mutator=seq mutator=SelectTarget(dev)
09:44:45 DEBUG Apply pid=57142 mutator=<func>
09:44:45 DEBUG Apply pid=57142 mutator=<func>
09:44:45 DEBUG Apply pid=57142 mutator=seq
09:44:45 DEBUG Apply pid=57142 mutator=seq mutator=initialize
09:44:45  INFO Phase: initialize pid=57142 mutator=seq mutator=initialize
09:44:45 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq
09:44:45 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=validate:AllResourcesHaveValues
09:44:45 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=RewriteSyncPaths
09:44:45 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=SyncDefaultPath
09:44:45 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=SyncInferRoot
09:44:45 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=MergeJobClusters
09:44:45 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=MergeJobParameters
09:44:45 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=MergeJobTasks
09:44:45 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=MergePipelineClusters
09:44:45 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=InitializeWorkspaceClient
09:44:45 DEBUG Loading profile DEFAULT because of host match pid=57142
09:44:45 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=PopulateCurrentUser
09:44:45 DEBUG Loading profile DEFAULT because of host match pid=57142
09:44:45  INFO Ignoring pat auth, because databricks-cli is preferred pid=57142 sdk=true
09:44:45  INFO Ignoring basic auth, because databricks-cli is preferred pid=57142 sdk=true
09:44:45  INFO Ignoring oauth-m2m auth, because databricks-cli is preferred pid=57142 sdk=true
09:44:45 DEBUG Running command: /opt/homebrew/Cellar/[email protected]/0.227.0/bin/databricks auth token --host https://adb-xxx-azuredatabricks.net pid=57142 sdk=true
09:44:46  INFO Refreshed OAuth token from Databricks CLI, expires on 2024-09-06 10:44:46.860721 -0400 EDT pid=57142 sdk=true
09:44:46 DEBUG Using Databricks CLI authentication with Databricks OAuth tokens pid=57142 sdk=true
09:44:46 DEBUG Running command: /opt/homebrew/Cellar/[email protected]/0.227.0/bin/databricks auth token --host https://adb-xxx-azuredatabricks.net pid=57142 sdk=true
09:44:46  INFO Refreshed OAuth token from Databricks CLI, expires on 2024-09-06 10:44:46.860721 -0400 EDT pid=57142 sdk=true
09:44:47 DEBUG GET /api/2.0/preview/scim/v2/Me
< HTTP/2.0 200 OK
< {
<   "active": true,
<   "displayName": "xxx",
<   "emails": [
<     {
<       "primary": true,
<       "type": "work",
<       "value": "[email protected]"
<     }
<   ],
<   "externalId": "037a9f7f-ec55-423b-a30f-4734a2603845",
<   "groups": [
<     {
<       "$ref": "Groups/778794051400239",
<       "display": xxx,
<       "type": "direct",
<       "value": "778794051400239"
<     }
<   ],
<   "id": "4286441873933971",
<   "name": {
<     "familyName": "xxx",
<     "givenName": "xxx"
<   },
<   "schemas": [
<     "urn:ietf:params:scim:schemas:core:2.0:User",
<     "urn:ietf:params:scim:schemas:extension:workspace:2.0:User"
<   ],
<   "userName": "[email protected]"
< } pid=57142 mutator=seq mutator=initialize mutator=seq mutator=PopulateCurrentUser sdk=true
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=DefineDefaultWorkspaceRoot
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=ExpandWorkspaceRoot
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=DefaultWorkspacePaths
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=SetVariables
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=ResolveVariableReferences
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=ResolveResourceReferences
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=ResolveVariableReferences
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=ResolveVariableReferences
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=SetRunAs
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=OverrideCompute
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=ProcessTargetMode
09:44:47  INFO Development mode: disabling deployment lock since bundle.deployment.lock.enabled is not set to true pid=57142 mutator=seq mutator=initialize mutator=seq mutator=ProcessTargetMode
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=ApplyPresets
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=DefaultQueueing
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=ExpandPipelineGlobPaths
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=ConfigureWSFS
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=TranslatePaths
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=PythonWrapperWarning
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=ApplyBundlePermissions
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=FilterCurrentUserFromPermissions
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=metadata.AnnotateJobs
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=metadata.AnnotatePipelines
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
09:44:47 DEBUG DATABRICKS_TF_EXEC_PATH is not defined pid=57142 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
09:44:47 DEBUG Using Terraform at /Users/xxx/git_repos/test_this/.databricks/bundle/dev/bin/terraform pid=57142 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
09:44:47 DEBUG DATABRICKS_TF_CLI_CONFIG_FILE is not defined pid=57142 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
09:44:47 DEBUG Environment variables for Terraform: DATABRICKS_AUTH_TYPE, HOME, PATH, TMPDIR, DATABRICKS_USER_AGENT_EXTRA, DATABRICKS_HOST pid=57142 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=initialize mutator=seq mutator=scripts.postinit
09:44:47 DEBUG No script defined for postinit, skipping pid=57142 mutator=seq mutator=initialize mutator=seq mutator=scripts.postinit
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build
09:44:47  INFO Phase: build pid=57142 mutator=seq mutator=build
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=scripts.prebuild
09:44:47 DEBUG No script defined for prebuild, skipping pid=57142 mutator=seq mutator=build mutator=seq mutator=scripts.prebuild
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.DetectPackages
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.DetectPackages mutator=seq
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.DetectPackages mutator=seq mutator=artifacts.whl.AutoDetect
09:44:47  INFO Detecting Python wheel project... pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.DetectPackages mutator=seq mutator=artifacts.whl.AutoDetect
09:44:47  INFO Found Python wheel project at /Users/xxx/git_repos/test_this pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.DetectPackages mutator=seq mutator=artifacts.whl.AutoDetect
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.inferAll
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.inferAll mutator=seq
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.inferAll mutator=seq mutator=artifacts.Infer(test_this)
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.inferAll mutator=seq mutator=artifacts.Infer(test_this) mutator=artifacts.whl.Infer(test_this)
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.PrepareAll
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.PrepareAll mutator=seq
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.PrepareAll mutator=seq mutator=artifacts.Prepare(test_this)
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.PrepareAll mutator=seq mutator=artifacts.Prepare(test_this) mutator=artifacts.whl.Prepare(test_this)
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq mutator=artifacts.Build(test_this)
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq mutator=artifacts.Build(test_this) mutator=seq
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq mutator=artifacts.Build(test_this) mutator=seq mutator=artifacts.whl.Build(test_this)
Building test_this...
09:44:47  INFO Build succeeded pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq mutator=artifacts.Build(test_this) mutator=seq mutator=artifacts.whl.Build(test_this)
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq mutator=artifacts.Build(test_this) mutator=seq mutator=artifacts.ExpandGlobs(test_this)
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=scripts.postbuild
09:44:47 DEBUG No script defined for postbuild, skipping pid=57142 mutator=seq mutator=build mutator=seq mutator=scripts.postbuild
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=build mutator=seq mutator=ResolveVariableReferences
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=deploy
09:44:47  INFO Phase: deploy pid=57142 mutator=seq mutator=deploy
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=deploy mutator=seq
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=scripts.predeploy
09:44:47 DEBUG No script defined for predeploy, skipping pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=scripts.predeploy
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=lock:acquire
09:44:47  INFO Skipping; locking is disabled pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=lock:acquire
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq
09:44:47 DEBUG Apply pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform:state-pull
09:44:49 DEBUG GET /api/2.0/workspace/get-status?path=/Users/[email protected]/.bundle/test_this/dev/state/terraform.tfstate&return_export_info=true
< HTTP/2.0 200 OK
< {
*****
< } pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform:state-pull sdk=true
09:44:50 DEBUG GET /api/2.0/workspace/export?direct_download=true&path=/Users/[email protected]/.bundle/test_this/dev/state/terraform.tfstate
< HTTP/2.0 200 OK
< <Streaming response> pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform:state-pull sdk=true
09:44:50 DEBUG Apply pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=deploy:state-pull
09:44:50  INFO Opening remote deployment state file pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=deploy:state-pull
09:44:51 DEBUG GET /api/2.0/workspace/get-status?path=/Users/[email protected]/.bundle/test_this/dev/state/deployment.json&return_export_info=true
< HTTP/2.0 200 OK
< {
<   "created_at": 1725583110251,
<   "modified_at": 1725626329939,
<   "object_id": 3405635552738227,
<   "object_type": "FILE",
<   "path": "/Users/[email protected]/.bundle/test_this/dev/state/deployment.json",
<   "resource_id": "xxx"
< } pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=deploy:state-pull sdk=true
09:44:51 DEBUG GET /api/2.0/workspace/export?direct_download=true&path=/Users/[email protected]/.bundle/test_this/dev/state/deployment.json
< HTTP/2.0 200 OK
< <Streaming response> pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=deploy:state-pull sdk=true
09:44:51  INFO Local deployment state is the same or newer, ignoring remote state pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=deploy:state-pull
09:44:51 DEBUG Apply pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=ValidateGitDetails
09:44:51 DEBUG Apply pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=artifacts.CleanUp
09:44:51 ERROR failed to delete /Users/[email protected]/.bundle/test_this/dev/artifacts/.internal: unable to delete filer root pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=artifacts.CleanUp
09:44:51 DEBUG POST /api/2.0/workspace/mkdirs
> {
>   "path": "/Users/[email protected]/.bundle/test_this/dev/artifacts/.internal"
> }
< HTTP/2.0 200 OK
< {} pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=artifacts.CleanUp sdk=true
09:44:51 DEBUG Apply pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=libraries.ExpandGlobReferences
09:44:51 ERROR Error: no files match pattern: test_this/dist/*.whl pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=libraries.ExpandGlobReferences
09:44:51 ERROR Error: no files match pattern: test_this/dist/*.whl pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq
09:44:51 DEBUG Apply pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=lock:release
09:44:51  INFO Skipping; locking is disabled pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=lock:release
09:44:51 ERROR Error: no files match pattern: test_this/dist/*.whl pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred
09:44:51 ERROR Error: no files match pattern: test_this/dist/*.whl pid=57142 mutator=seq mutator=deploy mutator=seq mutator=seq
09:44:51 ERROR Error: no files match pattern: test_this/dist/*.whl pid=57142 mutator=seq mutator=deploy mutator=seq
09:44:51 ERROR Error: no files match pattern: test_this/dist/*.whl pid=57142 mutator=seq mutator=deploy
09:44:51 ERROR Error: no files match pattern: test_this/dist/*.whl pid=57142 mutator=seq
Error: no files match pattern: test_this/dist/*.whl
  at resources.jobs.test_this_job.tasks[2].libraries[0].whl
  in resources/test_this_job.yml:39:15

09:44:51 ERROR failed execution pid=57142 exit_code=1 error=AlreadyPrinted````
@lachniej lachniej added the DABs DABs related issues label Sep 6, 2024
@pietern pietern mentioned this issue Sep 9, 2024
Merged
@pietern pietern added the Bug Something isn't working label Sep 9, 2024
@pietern pietern self-assigned this Sep 9, 2024
@pietern
Copy link
Contributor

pietern commented Sep 9, 2024

@lachniej Thanks for reporting the issue!

I didn't apply the relevant changes for library glob expansion as part of #1694 and that's where the first issue you list comes from. A fix for this issue is in #1756 and will be included in the next release later this week.

The PyDABs issue you mention (Error: build failed default, error: chdir) doesn't match the debug output.

Could you double check this?

github-merge-queue bot pushed a commit that referenced this issue Sep 9, 2024
@pietern
Expand library globs relative to the sync root (#1756)
b451905 
## Changes

Library glob expansion happens during deployment. Before that, all
entries that refer to local paths in resource definitions are made
relative to the _sync root_. Before #1694, they were made relative to
the _bundle root_. This PR didn't update the library glob expansion code
to use the sync root path.

If you were using the sync paths setting with library globs, the CLI
would fail to expand the globs because the code was using the wrong path
to anchor those globs.

This change fixes the issue.

## Tests

Manually confirmed that this fixes the issue reported in #1755.
@lachniej
Copy link
Author

lachniej commented Sep 9, 2024

@pietern thanks for that info.

Yeah sorry, I forgot to switch the version back to the supported databricks cli version for pydabs 5.1. Here is actual error.

10:24:54  INFO start pid=39810 version=0.228.0 args="databricks, bundle, deploy, --target, dev, --log-level=debug"
10:24:54 DEBUG Found bundle root at /Users/xxx/git_repos/pydabs5_1/pydabs_5_1 (file /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/databricks.yml) pid=39810
10:24:54 DEBUG Apply pid=39810 mutator=load
10:24:54  INFO Phase: load pid=39810 mutator=load
10:24:54 DEBUG Apply pid=39810 mutator=load mutator=seq
10:24:54 DEBUG Apply pid=39810 mutator=load mutator=seq mutator=EntryPoint
10:24:54 DEBUG Apply pid=39810 mutator=load mutator=seq mutator=scripts.preinit
Executing 'preinit' script
10:24:54 DEBUG Apply pid=39810 mutator=load mutator=seq mutator=ProcessRootIncludes
10:24:54 DEBUG Apply pid=39810 mutator=load mutator=seq mutator=ProcessRootIncludes mutator=seq
10:24:54 DEBUG Apply pid=39810 mutator=load mutator=seq mutator=VerifyCliVersion
10:24:54 DEBUG Apply pid=39810 mutator=load mutator=seq mutator=EnvironmentsToTargets
10:24:54 DEBUG Apply pid=39810 mutator=load mutator=seq mutator=InitializeVariables
10:24:54 DEBUG Apply pid=39810 mutator=load mutator=seq mutator=DefineDefaultTarget(default)
10:24:54 DEBUG Apply pid=39810 mutator=load mutator=seq mutator=LoadGitDetails
10:24:54 DEBUG Apply pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG running: .venv/bin/python3 -m databricks.bundles.build --phase load --input /var/folders/t1/3z0jrg357lx573rc76jjyhdw0000gq/T/-pydabs2277372783/input.json --output /var/folders/t1/3z0jrg357lx573rc76jjyhdw0000gq/T/-pydabs2277372783/output.json --diagnostics /var/folders/t1/3z0jrg357lx573rc76jjyhdw0000gq/T/-pydabs2277372783/diagnostics.json pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1' pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples' pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples.conditional_execution' pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Discovered job pydabs_5_1_examples_conditional_execution_detect_anomalies pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples.hello_world' pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Discovered job pydabs_5_1_examples_hello_world_hello_world pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples.library_dependencies' pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Discovered job pydabs_5_1_examples_library_dependencies_cowsay_job pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples.python_task' pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples.resource_generator' pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Discovered resource generator my_notebook_jobs pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples.task_values' pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Discovered job pydabs_5_1_examples_task_values_task_values pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.mutators' pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG stderr: DEBUG:root:Discovered resource mutator default_job_cluster_mutator pid=39810 mutator=load mutator=seq mutator=PythonMutator(load)
10:24:54 DEBUG Apply pid=39810 mutator=load mutator=seq mutator=validate:unique_resource_keys
10:24:54 DEBUG Apply pid=39810 mutator=load mutator=seq mutator=SelectTarget(dev)
10:24:54 DEBUG Apply pid=39810 mutator=<func>
10:24:54 DEBUG Apply pid=39810 mutator=<func>
10:24:54 DEBUG Apply pid=39810 mutator=seq
10:24:54 DEBUG Apply pid=39810 mutator=seq mutator=initialize
10:24:54  INFO Phase: initialize pid=39810 mutator=seq mutator=initialize
10:24:54 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq
10:24:54 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=validate:AllResourcesHaveValues
10:24:54 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=RewriteSyncPaths
10:24:54 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=SyncDefaultPath
10:24:54 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=SyncInferRoot
10:24:54 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=MergeJobClusters
10:24:54 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=MergeJobParameters
10:24:54 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=MergeJobTasks
10:24:54 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=MergePipelineClusters
10:24:54 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=InitializeWorkspaceClient
10:24:54 DEBUG Loading profile DEFAULT because of host match pid=39810
10:24:54 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PopulateCurrentUser
10:24:54 DEBUG Loading profile DEFAULT because of host match pid=39810
10:24:54  INFO Ignoring pat auth, because databricks-cli is preferred pid=39810 sdk=true
10:24:54  INFO Ignoring basic auth, because databricks-cli is preferred pid=39810 sdk=true
10:24:54  INFO Ignoring oauth-m2m auth, because databricks-cli is preferred pid=39810 sdk=true
10:24:54 DEBUG Running command: /opt/homebrew/Cellar/databricks/0.228.0/bin/databricks auth token --host https://adb-xxx.azuredatabricks.net pid=39810 sdk=true
10:24:54  INFO Refreshed OAuth token from Databricks CLI, expires on 2024-09-09 11:24:15.487025 -0400 EDT pid=39810 sdk=true
10:24:54 DEBUG Using Databricks CLI authentication with Databricks OAuth tokens pid=39810 sdk=true
10:24:54 DEBUG Running command: /opt/homebrew/Cellar/databricks/0.228.0/bin/databricks auth token --host https://adb-xxx.azuredatabricks.net pid=39810 sdk=true
10:24:54  INFO Refreshed OAuth token from Databricks CLI, expires on 2024-09-09 11:24:15.487025 -0400 EDT pid=39810 sdk=true
10:24:55 DEBUG GET /api/2.0/preview/scim/v2/Me
< HTTP/2.0 200 OK
< {
<   "active": true,
<   "displayName": "xxx",
<   "emails": [
<     {
<       "primary": true,
<       "type": "work",
<       "value": "xxx"
<     }
<   ],
<   "externalId": "037a9f7f-ec55-423b-a30f-4734a2603845",
<   "groups": [
<     {
<       "$ref": "Groups/778794051400239",
<       "display": "xxx",
<       "type": "direct",
<       "value": "778794051400239"
<     }
<   ],
<   "id": "4286441873933971",
<   "name": {
<     "familyName": "xxx",
<     "givenName": "xxx"
<   },
<   "schemas": [
<     "urn:ietf:params:scim:schemas:core:2.0:User",
<     "urn:ietf:params:scim:schemas:extension:workspace:2.0:User"
<   ],
<   "userName": "xxx"
< } pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PopulateCurrentUser sdk=true
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=DefineDefaultWorkspaceRoot
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=ExpandWorkspaceRoot
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=DefaultWorkspacePaths
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=SetVariables
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG running: .venv/bin/python3 -m databricks.bundles.build --phase init --input /var/folders/t1/3z0jrg357lx573rc76jjyhdw0000gq/T/-pydabs2355978857/input.json --output /var/folders/t1/3z0jrg357lx573rc76jjyhdw0000gq/T/-pydabs2355978857/output.json --diagnostics /var/folders/t1/3z0jrg357lx573rc76jjyhdw0000gq/T/-pydabs2355978857/diagnostics.json pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1' pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples' pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples.conditional_execution' pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Discovered job pydabs_5_1_examples_conditional_execution_detect_anomalies pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples.hello_world' pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Discovered job pydabs_5_1_examples_hello_world_hello_world pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples.library_dependencies' pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Discovered job pydabs_5_1_examples_library_dependencies_cowsay_job pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples.python_task' pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples.resource_generator' pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Discovered resource generator my_notebook_jobs pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.examples.task_values' pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Discovered job pydabs_5_1_examples_task_values_task_values pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Loading 'pydabs_5_1.mutators' pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG stderr: DEBUG:root:Discovered resource mutator default_job_cluster_mutator pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG Insert value at "resources.jobs.pydabs_5_1_examples_conditional_execution_detect_anomalies.job_clusters" pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG Insert value at "resources.jobs.pydabs_5_1_examples_conditional_execution_detect_anomalies.tasks[0].job_cluster_key" pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG Insert value at "resources.jobs.pydabs_5_1_examples_conditional_execution_detect_anomalies.tasks[2].job_cluster_key" pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG Insert value at "resources.jobs.pydabs_5_1_examples_hello_world_hello_world.job_clusters" pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG Insert value at "resources.jobs.pydabs_5_1_examples_hello_world_hello_world.tasks[0].job_cluster_key" pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG Insert value at "resources.jobs.pydabs_5_1_examples_library_dependencies_cowsay_job.job_clusters" pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG Insert value at "resources.jobs.pydabs_5_1_examples_library_dependencies_cowsay_job.tasks[0].job_cluster_key" pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG Insert value at "resources.jobs.pydabs_5_1_examples_task_values_task_values.job_clusters" pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG Insert value at "resources.jobs.pydabs_5_1_examples_task_values_task_values.tasks[0].job_cluster_key" pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG Insert value at "resources.jobs.pydabs_5_1_examples_task_values_task_values.tasks[1].job_cluster_key" pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG Insert value at "resources.jobs.notebook" pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonMutator(init)
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=ResolveVariableReferences
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=ResolveResourceReferences
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=ResolveVariableReferences
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=ResolveVariableReferences
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=SetRunAs
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=OverrideCompute
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=ProcessTargetMode
10:24:55  INFO Development mode: disabling deployment lock since bundle.deployment.lock.enabled is not set to true pid=39810 mutator=seq mutator=initialize mutator=seq mutator=ProcessTargetMode
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=ApplyPresets
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=DefaultQueueing
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=ExpandPipelineGlobPaths
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=ConfigureWSFS
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=TranslatePaths
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=PythonWrapperWarning
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=ApplyBundlePermissions
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=FilterCurrentUserFromPermissions
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=metadata.AnnotateJobs
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=metadata.AnnotatePipelines
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
10:24:55 DEBUG DATABRICKS_TF_EXEC_PATH is not defined pid=39810 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
10:24:55 DEBUG Using Terraform at /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/.databricks/bundle/dev/bin/terraform pid=39810 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
10:24:55 DEBUG DATABRICKS_TF_CLI_CONFIG_FILE is not defined pid=39810 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
10:24:55 DEBUG Environment variables for Terraform: DATABRICKS_USER_AGENT_EXTRA, DATABRICKS_HOST, DATABRICKS_AUTH_TYPE, HOME, PATH, TMPDIR pid=39810 mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=initialize mutator=seq mutator=scripts.postinit
10:24:55 DEBUG No script defined for postinit, skipping pid=39810 mutator=seq mutator=initialize mutator=seq mutator=scripts.postinit
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build
10:24:55  INFO Phase: build pid=39810 mutator=seq mutator=build
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=scripts.prebuild
10:24:55 DEBUG No script defined for prebuild, skipping pid=39810 mutator=seq mutator=build mutator=seq mutator=scripts.prebuild
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.DetectPackages
10:24:55 DEBUG artifacts block is defined, skipping auto-detecting pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.DetectPackages
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.inferAll
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.inferAll mutator=seq
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.inferAll mutator=seq mutator=artifacts.Infer(default)
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.PrepareAll
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.PrepareAll mutator=seq
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.PrepareAll mutator=seq mutator=artifacts.Prepare(default)
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.PrepareAll mutator=seq mutator=artifacts.Prepare(default) mutator=artifacts.whl.Prepare(default)
10:24:55 DEBUG open dir /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/pydabs_5_1: open /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/pydabs_5_1: no such file or directory pid=39810
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq mutator=artifacts.Build(default)
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq mutator=artifacts.Build(default) mutator=seq
10:24:55 DEBUG Apply pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq mutator=artifacts.Build(default) mutator=seq mutator=artifacts.whl.Build(default)
Building default...
10:24:55 ERROR Error: build failed default, error: chdir /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/pydabs_5_1: no such file or directory, output:  pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq mutator=artifacts.Build(default) mutator=seq mutator=artifacts.whl.Build(default)
10:24:55 ERROR Error: build failed default, error: chdir /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/pydabs_5_1: no such file or directory, output:  pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq mutator=artifacts.Build(default) mutator=seq
10:24:55 ERROR Error: build failed default, error: chdir /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/pydabs_5_1: no such file or directory, output:  pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq mutator=artifacts.Build(default)
10:24:55 ERROR Error: build failed default, error: chdir /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/pydabs_5_1: no such file or directory, output:  pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll mutator=seq
10:24:55 ERROR Error: build failed default, error: chdir /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/pydabs_5_1: no such file or directory, output:  pid=39810 mutator=seq mutator=build mutator=seq mutator=artifacts.BuildAll
10:24:55 ERROR Error: build failed default, error: chdir /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/pydabs_5_1: no such file or directory, output:  pid=39810 mutator=seq mutator=build mutator=seq
10:24:55 ERROR Error: build failed default, error: chdir /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/pydabs_5_1: no such file or directory, output:  pid=39810 mutator=seq mutator=build
10:24:55 ERROR Error: build failed default, error: chdir /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/pydabs_5_1: no such file or directory, output:  pid=39810 mutator=seq
Error: build failed default, error: chdir /Users/xxx/git_repos/pydabs5_1/pydabs_5_1/pydabs_5_1: no such file or directory, output:

@lachniej
Copy link
Author

lachniej commented Sep 13, 2024 •
edited
Loading

@pietern not sure if you had resolved what is going on with pydabs, but with the pre-release cli version right now dabs is working but pydabs is still returning this
image

@pietern
Copy link
Contributor

pietern commented Oct 24, 2024

@lachniej Could you confirm if the latest version of the CLI resolves the issue?

It should work for the regular DABs case. I'm not sure about the PyDABs case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pietern pietern

Labels
Bug Something isn't working DABs DABs related issues
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pietern @lachniej