Downloader doesn't handle query strings

[cigzigwon]

When trying to access files on a webserver it doesn't work: https://bcbsil.mrf.bcbs.com/2023-08_361_50I0_in-network-rates.json.gz?&Expires=1695132051&Signature=KLRMAB4vlXMcDEuTocPFwR00lEMbnzuGBNGeTcTcjyPBQVhGpUxx-CIQu6ePzTRDDvS7ZYroay-YDoeU3TaMcKl51m5OjF9BBzL2V33Kd8jXIcOZMU3fblygJcp4sr2359Ec05zdcKO6ecsHFNiV~lvGcNCezoNOIkzFR0aR2hB~VPas31Sfdplp6NiBgqflKzqBBaR~bbOLrb3U3thCE6ffSMI3r8xA4EbMWlxTE1fqnfLse-Y-tWL1V7SD7t3CFlLP~fKg~sCiqbyCMOLklfIg-R3HfUsl9r0RRpU9gvA3OCec9nQyOGmtipciVKJ-RTB-CS4HeiafXPojoSch8A__&Key-Pair-Id=K27TQMT39R1C8A

[cigzigwon]

GL-MO-LT-00004:bin csobeck$ docker run -it -v "$(pwd)/tmp/mrf-parse:/tmp/mrf-parse:rw" dancarbone/danielchalef-mrfparse pipeline -i "https://bcbsil.mrf.bcbs.com/2023-08_361_50I0_in-network-rates.json.gz?&Expires=1695132051&Signature=KLRMAB4vlXMcDEuTocPFwR00lEMbnzuGBNGeTcTcjyPBQVhGpUxx-CIQu6ePzTRDDvS7ZYroay-YDoeU3TaMcKl51m5OjF9BBzL2V33Kd8jXIcOZMU3fblygJcp4sr2359Ec05zdcKO6ecsHFNiV~lvGcNCezoNOIkzFR0aR2hB~VPas31Sfdplp6NiBgqflKzqBBaR~bbOLrb3U3thCE6ffSMI3r8xA4EbMWlxTE1fqnfLse-Y-tWL1V7SD7t3CFlLP~fKg~sCiqbyCMOLklfIg-R3HfUsl9r0RRpU9gvA3OCec9nQyOGmtipciVKJ-RTB-CS4HeiafXPojoSch8A__&Key-Pair-Id=K27TQMT39R1C8A" -o /tmp/mrf-parse/outputs -p -1 INFO[2023-08-06T20:40:56Z] Running step: Download ERRO[2023-08-06T20:40:57Z] Fatal error in /app/pkg/mrfparse/pipeline/steps.go#116: open /tmp/mrfparse3808262212/src/2023-08_361_50I0_in-network-rates.json.gz?&Expires=1695132051&Signature=KLRMAB4vlXMcDEuTocPFwR00lEMbnzuGBNGeTcTcjyPBQVhGpUxx-CIQu6ePzTRDDvS7ZYroay-YDoeU3TaMcKl51m5OjF9BBzL2V33Kd8jXIcOZMU3fblygJcp4sr2359Ec05zdcKO6ecsHFNiV~lvGcNCezoNOIkzFR0aR2hB~VPas31Sfdplp6NiBgqflKzqBBaR~bbOLrb3U3thCE6ffSMI3r8xA4EbMWlxTE1fqnfLse-Y-tWL1V7SD7t3CFlLP~fKg~sCiqbyCMOLklfIg-R3HfUsl9r0RRpU9gvA3OCec9nQyOGmtipciVKJ-RTB-CS4HeiafXPojoSch8A__&Key-Pair-Id=K27TQMT39R1C8A: file name too long

[cigzigwon]

This is moot since I can wield mrfpare split and mrfparse parse in my current pipeline "fetch" stage. It just becomes a file to process locally.

I appreciate the work done here that deals with extraneous providers. At this point we are gathering all billing codes.

Is it possible to disable the services code filter CSV?

[danielchalef]

You should be able to override the Service List filter here:

mrfparse/pkg/mrfparse/mrf/in_network_rates.go

Line 404 in 1781ef1

return bct, bc, ((bct == "HCPCS" || bct == "CPT") && serviceList.Contains(bc))

Note that it's been a long time since I looked at the code and the MRF files were something of a moving target back in December. You may unfortunately need to dig in to make sure the parser still functions correctly and does so for all payers.