Improve compilation speed for android build #166

sgnanase · 2023-10-12T14:10:15Z

FFTF python script used for flashfiles.zip in IVI. It uses ZipWrite python api to create it. This method is time consuming and adds several unwanted build steps that adds overhead. Like, flash.json generation, unpacking of target files, installer.cmd generation.

This fix, changes FFTF python to a normal FFTF bash script, which uses tar + pigz combo for faster flashfiles generation.

NOTE:
We deprecate flash.json as its used only by PFT & installer.cmd will be maintained in product config repo.

Tracked-On: OAM-112083

sysopenci · 2023-10-12T14:12:12Z

Android CI has started Engineering Build for this issue ,Please check the linked Tracked-On issue/Android CI Web for more details.

sysopenci · 2023-10-12T14:13:16Z

Android CI has started Engineering Build for this issue ,Please check the linked Tracked-On issue/Android CI Web for more details.

sysopenci · 2023-10-12T14:58:43Z

Android CI has started Engineering Build for this issue ,Please check the linked Tracked-On issue/Android CI Web for more details.

sysopenci · 2023-10-13T06:14:53Z

Android CI has completed Engineering Build for this issue, build is FAILURE. Please check the linked Tracked-On issue/Android CI Web for more details

sysopenci · 2023-10-13T10:32:32Z

Android CI has completed Engineering Build for this issue, build is FAILURE. Please check the linked Tracked-On issue/Android CI Web for more details

sysopenci · 2023-10-13T18:43:28Z

Android CI has completed Engineering Build for this issue, build is FAILURE. Please check the linked Tracked-On issue/Android CI Web for more details

sysopenci · 2023-10-25T08:03:14Z

Android CI has started Engineering Build for this issue ,Please check the linked Tracked-On issue/Android CI Web for more details.

yang8621 · 2023-10-25T15:11:41Z

tasks/flashfiles.mk

@@ -349,20 +366,20 @@ flashfiles: $(INTEL_FACTORY_FLASHFILES_TARGET) publish_gptimage_var publish_mkdi
 ifeq (,$(filter apollo_ivi blizzard_ivi celadon_ivi,$(TARGET_PRODUCT)))
 	@echo "Publishing Release files started"
 	$(hide) mkdir -p $(TOP)/pub/$(TARGET_PRODUCT)/$(TARGET_BUILD_VARIANT)/Release_Files
-	$(hide) cp -r $(PRODUCT_OUT)/*-flashfiles-*.zip $(TOP)/pub/$(TARGET_PRODUCT)/$(TARGET_BUILD_VARIANT)/Release_Files
+	$(hide) cp -r $(PRODUCT_OUT)/*-flashfiles-*.$(fn_compress_format) $(TOP)/pub/$(TARGET_PRODUCT)/$(TARGET_BUILD_VARIANT)/Release_Files
 	$(hide) cp -r $(PRODUCT_OUT)/scripts $(TOP)/pub/$(TARGET_PRODUCT)/$(TARGET_BUILD_VARIANT)/Release_Files
 	$(hide) cp -r vendor/intel/utils/host $(TOP)/pub/$(TARGET_PRODUCT)/$(TARGET_BUILD_VARIANT)/Release_Files
 	$(hide) mv $(TOP)/pub/$(TARGET_PRODUCT)/$(TARGET_BUILD_VARIANT)/Release_Files/host $(TOP)/pub/$(TARGET_PRODUCT)/$(TARGET_BUILD_VARIANT)/Release_Files/patches
 	$(hide) cp -r $(TOP)/pub/$(TARGET_PRODUCT)/$(TARGET_BUILD_VARIANT)/Release_Files/* $(TOP)
 ifneq (,$(wildcard vendor/intel/utils_vertical))
 ifneq (,$(wildcard vendor/intel/fw/keybox_provisioning))
 	@echo "vertical_keybox_provisioning included"


need to check LEGACY_FLASHZIP_TRUE here?

fn_compress_format is already decided based on LEGACY_FLASHZIP_TRUE flag.

ifeq ($(LEGACY_FLASHZIP_TRUE),true)
fn_compress_format := zip
else
fn_compress_format := tar.gz
endif

We dont need to add another condition check for this pls.

yang8621 · 2023-10-25T15:13:35Z

tasks/flashfiles.mk

@@ -286,7 +303,7 @@ ISO_INSTALL_IMG = $(PRODUCT_OUT)/$(TARGET_PRODUCT)-sign-flashfile-$(FILE_NAME_TA
 else
 ISO_INSTALL_IMG = $(PRODUCT_OUT)/$(TARGET_PRODUCT)-flashfile-$(FILE_NAME_TAG).iso
 endif
-ISO_INSTALL_IMG_ZIP = $(ISO_INSTALL_IMG).zip
+ISO_INSTALL_IMG_ZIP = $(ISO_INSTALL_IMG).tar.gz


better to rename ISO_INSTALL_IMG_ZIP to ISO_INSTALL_IMG_TAR_GZ

I will make this change (y)

sysopenci · 2023-10-25T15:17:06Z

Android CI has completed Engineering Build for this issue, build is SUCCESS.To merge the changes please click on APPROVE FOR MERGE button in Android CI WEB

yang8621

Can we have a test:
rename /usr/bin/pigz to /usr/bin/pigz_bak
make flashfiles LEGACY_FLASHZIP_TRUE -jxx
Not sure pigz is Ubuntu built-in or manually installed? If it's manually installed, it can be added into env setup wiki.

sgnanase · 2023-10-26T13:40:13Z

Can we have a test:
rename /usr/bin/pigz to /usr/bin/pigz_bak
make flashfiles LEGACY_FLASHZIP_TRUE -jxx

Not sure pigz is Ubuntu built-in or manually installed? If it's manually installed, it can be added into env setup wiki.

By default, for ISO we are moving to tar+pigz from zip, So we need /usr/bin/pigz by default. We will capture this as pre-requisite.

sysopenci

PR integrated into 'preintegration' builder

More details, logs, and binaries: /absp/builders/celadon_smr0_master-preintegration/builds/857

Done by "Gnanasekaran, Sundar" [email protected]

sysopenci · 2023-10-27T07:49:36Z

Android CI has completed Engineering Build for this issue, build is FAILURE. Please check the linked Tracked-On issue/Android CI Web for more details

asaddamhu · 2023-11-07T15:23:04Z

@sgnanase , please don't deprecate flash.json as it is powerful, supports multi configuration flashing and steps.
We have created a simple tool to flash android(replacement for PFT) using 'flash.json' and 'installer.cmd'. Please consider this as a requirement to keep and maintain flash.json.

Thanks!

Currently FFTF python script is used for creating flashfiles.zip. It uses ZipWrite python api which is time consuming and adds overhead to build time. Say, flash.json generation, unpacking of target files, installer.cmd generation, etc. This fix helps to optimize build time and involves below changes: - Use tar + pigz combo for faster flashfiles generation. - Make FFTF a bash script as python doesn't directly support tar + pigz - This new bash script will support old python FFTF. Just pass LEGACY_FLASHZIP_TRUE=true flag to "make flashfiles" to enable it. - The new script still supports all old functionalities, say, generation of installer.cmd, flash.json, repacking of bootloader.img, etc. Tracked-On: OAM-112083 Signed-off-by: sgnanase <[email protected]>

GangSecurity · 2023-11-23T12:48:45Z

releasetools/flashfiles_from_target_files.sh

+echo "Generate installer.cmd"
+echo "========================"
+device/intel/build/releasetools/flash_cmd_generator.py device/intel/project-celadon/$TARGET/flashfiles.ini $TARGET $VARIANT | sed '$d' | sed '$d' | sed -n '/installer.cmd/,$p' | sed '1d' > $PRODUCT_OUT/$flashfile_dir/installer.cmd
+sed -i 's/flash super super.img/flash super super.img.part00 super.img.part01/g' $PRODUCT_OUT/$flashfile_dir/installer.cmd


if image is smaller than 4G, there is only super.img. Ff image is bigger than 4G, there will be super.img.part00, super.img.part01, if image is bigger then 8G, there will be super.img.part00, super.img.part01, super.img.part02...
You can decide refine the patch or submit another patch to fix.

One more comment is that flash.json will be used by new flash tool, did you keep it?

sgnanase force-pushed the speed branch from b0235a5 to 3702f8f Compare October 12, 2023 14:11

sysopenci added the Valid commit message label Oct 12, 2023

sgnanase force-pushed the speed branch from 3702f8f to 1870d5f Compare October 12, 2023 14:56

sysopenci added Valid commit message and removed Valid commit message labels Oct 12, 2023

sysopenci added Valid commit message and removed Valid commit message labels Oct 25, 2023

yang8621 reviewed Oct 25, 2023

View reviewed changes

GangSecurity approved these changes Oct 26, 2023

View reviewed changes

sysopenci added Valid commit message and removed Valid commit message labels Oct 26, 2023

sysopenci requested review from balajim001, bing8deng, JeevakaPrabu, kaushlen, lchen43 and rajucm2007 October 26, 2023 13:53

sysopenci requested review from balajim001, bing8deng, JeevakaPrabu, kaushlen, lchen43, rajucm2007, shyjumon-n, swaroopbalan, yyao3 and zhy1030 and removed request for swaroopbalan, shyjumon-n, balajim001, rajucm2007 and kaushlen October 27, 2023 02:36

sysopenci reviewed Oct 27, 2023

View reviewed changes

sysopenci added Valid commit message and removed Valid commit message PR approved labels Nov 22, 2023

sgnanase force-pushed the speed branch from 659084b to 6653ba4 Compare November 23, 2023 03:00

sysopenci added Valid commit message and removed Valid commit message labels Nov 23, 2023

yang8621 approved these changes Nov 23, 2023

View reviewed changes

GangSecurity approved these changes Nov 23, 2023

View reviewed changes

sysopenci added Valid commit message and removed Valid commit message labels Nov 23, 2023

sysopenci added the Stale Stale label for inactive open prs label Sep 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve compilation speed for android build #166

Improve compilation speed for android build #166

sgnanase commented Oct 12, 2023

sysopenci commented Oct 12, 2023

sysopenci commented Oct 12, 2023

sysopenci commented Oct 12, 2023

sysopenci commented Oct 13, 2023

sysopenci commented Oct 13, 2023

sysopenci commented Oct 13, 2023

sysopenci commented Oct 25, 2023

yang8621 Oct 25, 2023

sgnanase Oct 26, 2023

yang8621 Oct 25, 2023

sgnanase Oct 26, 2023

sysopenci commented Oct 25, 2023

yang8621 left a comment

sgnanase commented Oct 26, 2023

sysopenci left a comment

sysopenci commented Oct 27, 2023

asaddamhu commented Nov 7, 2023

GangSecurity Nov 23, 2023

GangSecurity Nov 23, 2023

Improve compilation speed for android build #166

Are you sure you want to change the base?

Improve compilation speed for android build #166

Conversation

sgnanase commented Oct 12, 2023

sysopenci commented Oct 12, 2023

sysopenci commented Oct 12, 2023

sysopenci commented Oct 12, 2023

sysopenci commented Oct 13, 2023

sysopenci commented Oct 13, 2023

sysopenci commented Oct 13, 2023

sysopenci commented Oct 25, 2023

yang8621 Oct 25, 2023

Choose a reason for hiding this comment

sgnanase Oct 26, 2023

Choose a reason for hiding this comment

yang8621 Oct 25, 2023

Choose a reason for hiding this comment

sgnanase Oct 26, 2023

Choose a reason for hiding this comment

sysopenci commented Oct 25, 2023

yang8621 left a comment

Choose a reason for hiding this comment

sgnanase commented Oct 26, 2023

sysopenci left a comment

Choose a reason for hiding this comment

sysopenci commented Oct 27, 2023

asaddamhu commented Nov 7, 2023

GangSecurity Nov 23, 2023

Choose a reason for hiding this comment

GangSecurity Nov 23, 2023

Choose a reason for hiding this comment