Looking for guidance of how to use another object_store with datafusion

[thhapke]

Sorry for this most basic question, but I do not get a grip on how to use an ObjectStore with datafusion to register tables, parquet files etc.

For my object store SAP HDLFS I have implemented the object_store::ObjectStore trait and an object_store registry

Now I like to load parquet and later datalake tables into a datafusion DataFrame, but with this code I run into an error

NotFound { path: "/DE/customer/0-0d3875e9-1611-4ccd-a801-184caf851de2-0.parquet", source: Os { code: 2, kind: NotFound, message: "No such file or directory" }

let ctx: SessionContext = SessionContext::new();
  let hdlfs_url = Url::parse("hdlfs://").context(ParseURLSnafu{})?;
  let hdlregistry = registry::HDLFSStoreProvider::new(&HDLFS_CONFIG);
  let hdlstore = hdlregistry.get_store(&hdlfs_url)
          .context(ObjectStoreNotFoundSnafu{object_store: hdlfs_url.to_string()})?;

  let table_name = "decustomer";
  let hdlfs_path = "/DE/customer/my.parquet";

  ctx.runtime_env().register_object_store(&hdlfs_url, Arc::new(hdlstore));
  ctx.register_parquet(table_name, hdlfs_path, ParquetReadOptions::default()).await.context(DataFusionSnafu{})?;

But with e.g.

let data = hdlstore.get_opts(&Path::from(hdlfs_path), GetOptions::default()).await
                             .context(DownloadSnafu{file:hdlfs_path.clone()})?;

Can anyone guide me into the right direction? My ultimate goal is to test datafusion against Spark and evaluate where we can complement Spark or replace Spark in specific contexts.

[thhapke]

As assumed it was a simple error. When I was calling:

    let b = ListingTableUrl::parse(hdlfs_path).context(DataFusionSnafu)?;
    println!("{:?}",b);

I found out that the correct path needs to be "hdlfs:///DE/customer/my.parquet", otherwise "DE" was interpreted as an host.