buildenv: using an forked openwrt repo and branch from github fails #780

pmelange · 2020-04-09T09:17:01Z

I would like to use a forked repo and a test branch to compile a test version of openwrt. Unfortunately it does not work.

Here are the errors

rm -f /home/pmelange/src/mt7621/openwrt/.config                                
rm -f .stamp-
rm -f .stamp-gluon-module-openwrt
echo >>.stamp-gluon-module-openwrt "OPENWRT_REPO=https://github.com/pmelange/openwrt.git"
echo >>.stamp-gluon-module-openwrt "OPENWRT_COMMIT=641703b11b7b4e0e17298bbe5386ae302d9ed955"
git clone $(grep _REPO .stamp-gluon-module-openwrt | cut -d "=" -f 2) /home/pmelange/src/mt7621/tmp/gluon_.stamp-gluon-module-openwrt                          
Cloning into '/home/pmelange/src/mt7621/tmp/gluon_.stamp-gluon-module-openwrt'...
remote: Enumerating objects: 32, done.
remote: Counting objects: 100% (32/32), done.
remote: Compressing objects: 100% (24/24), done.
remote: Total 504820 (delta 11), reused 16 (delta 8), pack-reused 504788
Receiving objects: 100% (504820/504820), 175.36 MiB | 4.92 MiB/s, done.        
Resolving deltas: 100% (344264/344264), done.                                  
cd /home/pmelange/src/mt7621/tmp/gluon_.stamp-gluon-module-openwrt; git name-rev --refs openwrt-* $(grep _COMMIT /home/pmelange/src/mt7621/.stamp-gluon-module-openwrt | cut -d "=" -f 2) | cut -d / -f 2 | cut -d \~ -f 1 >branchname.txt     
cd /home/pmelange/src/mt7621/tmp/gluon_.stamp-gluon-module-openwrt; grep -q undefined branchname.txt  || \
        printf >>/home/pmelange/src/mt7621/.stamp-gluon-module-openwrt "OPENWRT_BRANCH=%s\n" \
                $(echo  | tr '[:lower:]' '[:upper:]') \                        
                $(cat branchname.txt)
rm -rf /home/pmelange/src/mt7621/tmp/gluon_.stamp-gluon-module-openwrt         
rm -f /home/pmelange/src/mt7621/modules
cat >>/home/pmelange/src/mt7621/modules .stamp-gluon-module-openwrt            
cat >>/home/pmelange/src/mt7621/modules .stamp-gluon-module-packages .stamp-gluon-module-luci .stamp-gluon-module-freifunk .stamp-gluon-module-routing .stamp-gluon-module-packages_berlin .stamp-gluon-module-gluon                           
echo >>/home/pmelange/src/mt7621/modules GLUON_FEEDS=\' packages luci freifunk routing packages_berlin gluon  \'
--- Updating module 'openwrt' ---
Initialized empty Git repository in /home/pmelange/src/mt7621/openwrt/.git/    
remote: Enumerating objects: 10, done.
remote: Counting objects: 100% (10/10), done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 487662 (delta 2), reused 2 (delta 0), pack-reused 487652
Receiving objects: 100% (487662/487662), 171.07 MiB | 5.01 MiB/s, done.        
Resolving deltas: 100% (331972/331972), done.                                  
From https://github.com/pmelange/openwrt                                       
 * branch            HEAD       -> FETCH_HEAD                                  
fatal: Not a valid branch point: '641703b11b7b4e0e17298bbe5386ae302d9ed955'.   
unable to find commit "641703b11b7b4e0e17298bbe5386ae302d9ed955" on branch "" in repo "https://github.com/pmelange/openwrt.git".                               
make: *** [Makefile:136: .stamp-pre-patch] Error 1

If I use a commit ID which is on the master branch of my forked repo, it works, but if the commit is in a branch, I get the errors above.

I originally tried using vDorst/openwrt@70188cf and it failed with the same kind of error as above. So I added vDorst as a new remote, fetched the changes, cherry-picked into a new branch and tried from my forked repo.

I would really like a fix to this problem so that I can continue to develop and test.

@SvenRoederer you have made a lot of changes to the build environment. Can you please help?

The text was updated successfully, but these errors were encountered:

pmelange · 2020-04-09T09:22:35Z

I have merged the commits in the branch to my master, just so that I can continue with testing. It's working for now, but it shouldn't have to be done this way

pmelange · 2020-04-09T09:33:49Z

If you want to try this at home,

git checkout daily/upstream-master

in config.mk

...
OPENWRT_SRC=https://github.com/vDorst/openwrt.git
OPENWRT_COMMIT=70188cf0837e7991a8eea390fbd75d89062941b3
...

make TARGET=ramips-mt7621

sarumpaet · 2020-04-09T12:31:33Z

If you set OPENWRT_BRANCH=ramips-5.4 in config.mk as well, it works. There's some magic in our Makefile that looks like it should be doing that, but obviously it's broken (it surely looks very brittle anyways - I'd put such stuff outside of a Makefile as formatting and escaping will bite you).

pmelange · 2020-04-09T13:01:13Z

It looks like it should.... :/

$ cat config.mk

# default parameters for Makefile
SHELL:=$(shell which bash)
TARGET=ath79-generic
PACKAGES_LIST_DEFAULT=notunnel tunnel-berlin-tunneldigger manual
OPENWRT_SRC=https://github.com/vDorst/openwrt.git
OPENWRT_BRANCH=ramips-5.4
OPENWRT_COMMIT=70188cf0837e7991a8eea390fbd75d89062941b3
SET_BUILDBOT=env
MAKE_ARGS=

remote: Counting objects: 100% (2/2), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 496077 (delta 0), reused 0 (delta 0), pack-reused 496075
Receiving objects: 100% (496077/496077), 173.01 MiB | 2.54 MiB/s, done.        
Resolving deltas: 100% (337983/337983), done.                                  
cd /home/pmelange/src/mt7621/tmp/gluon_.stamp-gluon-module-openwrt; git name-rev --refs openwrt-* $(grep _COMMIT /home/pmelange/src/mt7621/.stamp-gluon-module-openwrt | cut -d "=" -f 2) | cut -d / -f 2 | cut -d \~ -f 1 >branchname.txt     
cd /home/pmelange/src/mt7621/tmp/gluon_.stamp-gluon-module-openwrt; grep -q undefined branchname.txt  || \
        printf >>/home/pmelange/src/mt7621/.stamp-gluon-module-openwrt "OPENWRT_BRANCH=%s\n" \
                $(echo  | tr '[:lower:]' '[:upper:]') \                        
                $(cat branchname.txt)
rm -rf /home/pmelange/src/mt7621/tmp/gluon_.stamp-gluon-module-openwrt         
rm -f /home/pmelange/src/mt7621/modules
cat >>/home/pmelange/src/mt7621/modules .stamp-gluon-module-openwrt            
cat >>/home/pmelange/src/mt7621/modules .stamp-gluon-module-packages .stamp-gluon-module-luci .stamp-gluon-module-freifunk .stamp-gluon-module-routing .stamp-gluon-module-packages_berlin .stamp-gluon-module-gluon                           
echo >>/home/pmelange/src/mt7621/modules GLUON_FEEDS=\' packages luci freifunk routing packages_berlin gluon  \'
--- Updating module 'openwrt' ---
Reinitialized existing Git repository in /home/pmelange/src/mt7621/openwrt/.git/
From https://github.com/vDorst/openwrt
 * branch                  HEAD       -> FETCH_HEAD                            
fatal: Not a valid branch point: '70188cf0837e7991a8eea390fbd75d89062941b3'.   
unable to find commit "70188cf0837e7991a8eea390fbd75d89062941b3" on branch "" in repo "https://github.com/vDorst/openwrt.git".                                 
make: *** [Makefile:136: .stamp-pre-patch] Error 1

sarumpaet · 2020-04-09T14:07:56Z

Weird - that fixed it for me. You aren't using a Mac by chance? Perhaps add some debug echo lines to scripts/update.sh to do tracing.

pmelange · 2020-04-09T14:43:27Z

No, I'm using linux on the machine where I build everything else. I'll take a look at it sometime, but not today.

pmelange · 2020-04-09T15:42:21Z

The branch "ramips-5.4" has been updated. I tried it again with @sarumpaet's suggestion, and it didn't work. :( Same error as last time.

--- Updating module 'openwrt' ---
Initialized empty Git repository in /home/pmelange/src/mt7621/openwrt/.git/    
remote: Enumerating objects: 2, done.
remote: Counting objects: 100% (2/2), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 487092 (delta 0), reused 0 (delta 0), pack-reused 487090
Receiving objects: 100% (487092/487092), 170.58 MiB | 6.61 MiB/s, done.        
Resolving deltas: 100% (331623/331623), done.                                  
From https://github.com/vDorst/openwrt
 * branch            HEAD       -> FETCH_HEAD                                  
fatal: Not a valid branch point: '9e59838fc68b75d2849ea7ba81de8869c71cd2df'.   
unable to find commit "9e59838fc68b75d2849ea7ba81de8869c71cd2df" on branch "" in repo "https://github.com/vDorst/openwrt.git".                                 
make: *** [Makefile:136: .stamp-pre-patch] Error 1

SvenRoederer · 2020-04-09T22:41:22Z

@pmelange

firmware/Makefile

Line 84 in b71cbcc

    
           cd $(GLUON_TMPDIR)/gluon_$@; git name-rev --refs openwrt-* $$(grep _COMMIT $(FW_DIR)/$@ | \

should be the source of the problem you are reporting. It's trying to find a branchname where this commits belong to and stores this in the branchname.txt file. This code has been primarily tested on official OpenWrt-branches only.

As mentioned by @sarumpaet the code added in 7a72a84 (PR #714) to generate the modules file (needed by the gluon-build) from our current config.mk and feeds.conf is nothing to be proud of.

SvenRoederer · 2020-04-09T23:15:05Z

I had a look at the referenced code of the repo of @vDorst.
These commits can also just copied (git format-patch) into patches/openwrt/ and will be applied onto the upstream-repo defined in config.mk.
Example: https://github.com/freifunk-berlin/firmware/blob/7a6f8cffa7f9caf8a10641bc7ee2a9a4a0091c62/patches/packages/freifunk/0002-PR18.patch

SvenRoederer · 2020-04-10T07:11:41Z

PR #781 will fix this problem by avoiding the "very brittle" code.

pmelange · 2020-04-10T08:30:59Z

I'll give PR #781 a try.

Switch from defining the upstream-repos and relating commit in config.mk and feeds.conf to define all in the modules-file. The modules-files is the real file used by the build-system to setup the working checkout. It originates from the Gluon-buildsystemand some documentation is avail at https://gluon.readthedocs.io/en/latest/dev/build.html This also relates to * PR freifunk-berlin#781 * Issue freifunk-berlin#780 * Issue freifunk-berlin#772

When using the modules-file in place of config.mk and feeds.conf to define the upstream-repos, there is no need to keep the code which will synthesize it. This primarily reverts 7a72a84. This also relates to * PR freifunk-berlin#781 * Issue freifunk-berlin#780 * Issue freifunk-berlin#772

Switch from defining the upstream-repos and relating commit in config.mk and feeds.conf to define all in the modules-file. The modules-files is the real file used by the build-system to setup the working checkout. It originates from the Gluon-buildsystemand some documentation is avail at https://gluon.readthedocs.io/en/latest/dev/build.html This also relates to * PR freifunk-berlin#781 * Issue freifunk-berlin#780 * Issue freifunk-berlin#772

When using the modules-file in place of config.mk and feeds.conf to define the upstream-repos, there is no need to keep the code which will synthesize it. This primarily reverts 7a72a84. This also relates to * PR freifunk-berlin#781 * Issue freifunk-berlin#780 * Issue freifunk-berlin#772

Switch from defining the upstream-repos and relating commit in config.mk and feeds.conf to define all in the modules-file. The modules-files is the real file used by the build-system to setup the working checkout. It originates from the Gluon-buildsystemand some documentation is avail at https://gluon.readthedocs.io/en/latest/dev/build.html This also relates to * PR #781 * Issue #780 * Issue #772

When using the modules-file in place of config.mk and feeds.conf to define the upstream-repos, there is no need to keep the code which will synthesize it. This primarily reverts 7a72a84. This also relates to * PR #781 * Issue #780 * Issue #772

Switch from defining the upstream-repos and relating commit in config.mk and feeds.conf to define all in the modules-file. The modules-files is the real file used by the build-system to setup the working checkout. It originates from the Gluon-buildsystemand some documentation is avail at https://gluon.readthedocs.io/en/latest/dev/build.html This also relates to * PR #781 * Issue #780 * Issue #772

When using the modules-file in place of config.mk and feeds.conf to define the upstream-repos, there is no need to keep the code which will synthesize it. This primarily reverts 7a72a84. This also relates to * PR #781 * Issue #780 * Issue #772

SvenRoederer · 2020-05-08T09:08:46Z

Should be fixed with merging PR #781.
Feel free to reopen if you still cant use other repos.

Switch from defining the upstream-repos and relating commit in config.mk and feeds.conf to define all in the modules-file. The modules-files is the real file used by the build-system to setup the working checkout. It originates from the Gluon-buildsystemand some documentation is avail at https://gluon.readthedocs.io/en/latest/dev/build.html This also relates to * PR freifunk-berlin#781 * Issue freifunk-berlin#780 * Issue freifunk-berlin#772

When using the modules-file in place of config.mk and feeds.conf to define the upstream-repos, there is no need to keep the code which will synthesize it. This primarily reverts 7a72a84. This also relates to * PR freifunk-berlin#781 * Issue freifunk-berlin#780 * Issue freifunk-berlin#772

Switch from defining the upstream-repos and relating commit in config.mk and feeds.conf to define all in the modules-file. The modules-files is the real file used by the build-system to setup the working checkout. It originates from the Gluon-buildsystemand some documentation is avail at https://gluon.readthedocs.io/en/latest/dev/build.html This also relates to * PR #781 * Issue #780 * Issue #772

When using the modules-file in place of config.mk and feeds.conf to define the upstream-repos, there is no need to keep the code which will synthesize it. This primarily reverts 7a72a84. This also relates to * PR #781 * Issue #780 * Issue #772

Switch from defining the upstream-repos and relating commit in config.mk and feeds.conf to define all in the modules-file. The modules-files is the real file used by the build-system to setup the working checkout. It originates from the Gluon-buildsystemand some documentation is avail at https://gluon.readthedocs.io/en/latest/dev/build.html This also relates to * PR freifunk-berlin#781 * Issue freifunk-berlin#780 * Issue freifunk-berlin#772

When using the modules-file in place of config.mk and feeds.conf to define the upstream-repos, there is no need to keep the code which will synthesize it. This primarily reverts 7a72a84. This also relates to * PR freifunk-berlin#781 * Issue freifunk-berlin#780 * Issue freifunk-berlin#772

This was referenced Apr 9, 2020

Using prebuilt OpenWRT stable releases for our Firmware #774

Closed

Use modules-file instead of generating it from config.mk and feeds.conf #781

Closed

SvenRoederer mentioned this issue Apr 10, 2020

buildenv: using HEAD of a feed will not work #777

Open

SvenRoederer added bug buildsystem labels Apr 10, 2020

SvenRoederer added this to the 2020.02 - Release milestone Apr 10, 2020

SvenRoederer closed this as completed May 8, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

buildenv: using an forked openwrt repo and branch from github fails #780

buildenv: using an forked openwrt repo and branch from github fails #780

pmelange commented Apr 9, 2020

pmelange commented Apr 9, 2020 •

edited

Loading

pmelange commented Apr 9, 2020

sarumpaet commented Apr 9, 2020

pmelange commented Apr 9, 2020

sarumpaet commented Apr 9, 2020

pmelange commented Apr 9, 2020

pmelange commented Apr 9, 2020

SvenRoederer commented Apr 9, 2020

SvenRoederer commented Apr 9, 2020 •

edited

Loading

SvenRoederer commented Apr 10, 2020

pmelange commented Apr 10, 2020

SvenRoederer commented May 8, 2020

buildenv: using an forked openwrt repo and branch from github fails #780

buildenv: using an forked openwrt repo and branch from github fails #780

Comments

pmelange commented Apr 9, 2020

pmelange commented Apr 9, 2020 • edited Loading

pmelange commented Apr 9, 2020

sarumpaet commented Apr 9, 2020

pmelange commented Apr 9, 2020

sarumpaet commented Apr 9, 2020

pmelange commented Apr 9, 2020

pmelange commented Apr 9, 2020

SvenRoederer commented Apr 9, 2020

SvenRoederer commented Apr 9, 2020 • edited Loading

SvenRoederer commented Apr 10, 2020

pmelange commented Apr 10, 2020

SvenRoederer commented May 8, 2020

pmelange commented Apr 9, 2020 •

edited

Loading

SvenRoederer commented Apr 9, 2020 •

edited

Loading