Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DB Null Exception #292

Open
RKvign01 opened this issue Jul 27, 2023 · 3 comments
Open

DB Null Exception #292

RKvign01 opened this issue Jul 27, 2023 · 3 comments
Labels

Comments

@RKvign01
Copy link

Hi,

            SqlConnection conn = new SqlConnection(StreamExtensions.getConnectionString());
            conn.Open();
            SqlCommand cmd = new SqlCommand(queryDB, conn);

            using (var reader = cmd.ExecuteReader())
            {

                using (var parser = new ChoParquetWriter("test.parquet")
                .Configure(c => c.CompressionMethod = CompressionMethod.Snappy)
                .Configure(c => c.LiteParsing = false)
                .Configure(c => c.RowGroupSize = 20)
                .NotifyAfter(1000)
                .OnRowsWritten((o, e) => $"Rows: {e.RowsWritten} <--- {DateTime.Now}".Print()))
                {
                    if (reader.HasRows)
                    {
                        parser.Write(reader);
                    }
                }
            }

            conn.Close();
            
            
           I am using the above code to perform a dbraeder to parquet conversion but I am facing below cast issue,
image
@RKvign01
Copy link
Author

RKvign01 commented Jul 27, 2023

Please find the version I am using,

image

@Cinchoo
Copy link
Owner

Cinchoo commented Aug 4, 2023

Here is one way to handle DbNull values by subscribing to BeforeRecordFieldWrite event and process such values.

        using (var r = command.ExecuteReader(CommandBehavior.CloseConnection))
        {
            using (var parser = new ChoParquetWriter(filePath)
                .Configure(c => c.CompressionMethod = Parquet.CompressionMethod.Gzip)
                .Configure(c => c.RowGroupSize = 1000)
                .NotifyAfter(1000)
                .OnRowsWritten((o, e) => $"Rows: {e.RowsWritten} <--- {DateTime.Now}".Print())
                .Setup(s => s.BeforeRecordFieldWrite += (o, e) =>
                {
                    if (e.Source == DBNull.Value)
                        e.Source = null;
                })
                )
            {
                if (r.HasRows)
                {
                    parser.Write(r);
                }
            }
        }

@Cinchoo
Copy link
Owner

Cinchoo commented Aug 8, 2023

With the latest release v1.2.1.61, this will be handled by the framework automatically, no need to use BeforeRecordFieldWrite event.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants