buildscript/patches: Automatically scan directories for feeds

Submitted by Adrian Schmutzler on April 3, 2019, 12:53 p.m.

Details

Message ID 20190403125336.3145-1-freifunk@adrianschmutzler.de
State Superseded
Headers show

Commit Message

Adrian Schmutzler April 3, 2019, 12:53 p.m.
The buildscript knows two different types of patches, which are
applied to pulled-in repositories:

1. Feed patches
Those are applied as "GIT patches" to the relevant repos,
directly after those have been checked out.
They reside in subfolders of the build_patches folder, and
have to be selected individually and manually in the
buildscript.sh.

2. Build patches
Those are applied later in the process, just using the system
patch tool, and changing the $target directory.
All patches in the folder "build_patches/openwrt" are read
and applied automatically.

This is both inconsistent (two different types of patches in
the same dir) and annoying (feed patches have to be specified
by hand), especially for unexperienced developers.

This patch addresses this by:
- separating files into two dirs: build_patches and feed_patches
- automatically scanning feed patches and thus having similar
  experience for the user (I cannot think of a case where we
  provide a patch, but do not use it)

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
---
 buildscript                                        | 34 +++++++++-------------
 ...se-hotplug.d-iface-instead-of-hotplug.d-n.patch |  0
 .../0020-fastd_generate_key_from_urandom.patch     |  0
 ...adv-patch-to-remove-gw-mode-switch-messag.patch |  0
 4 files changed, 14 insertions(+), 20 deletions(-)
 rename {build_patches => feed_patches}/gluon/0001-simple-tc-Use-hotplug.d-iface-instead-of-hotplug.d-n.patch (100%)
 rename {build_patches/openwrt/fastd => feed_patches/openwrt}/0020-fastd_generate_key_from_urandom.patch (100%)
 rename {build_patches => feed_patches}/routing/0002-Add-batman-adv-patch-to-remove-gw-mode-switch-messag.patch (100%)

Patch hide | download patch | download mbox

diff --git a/buildscript b/buildscript
index 1d527c6e..a6725c5d 100755
--- a/buildscript
+++ b/buildscript
@@ -23,23 +23,20 @@  PACKAGEURL="https://git.openwrt.org/feed/packages.git"
 #official openwrt packages
 OPENWRT=(openwrt
          $PACKAGEURL
-         $PACKAGEREV
-         fastd/0020-fastd_generate_key_from_urandom.patch)
+         $PACKAGEREV)
 OPENWRT_PKGS="gpioctl-sysfs libugpio fastd haserl"
 
 ## Be careful: FFF uses COMPAT_VERSION 15 as default at the moment.
 ## See http://www.open-mesh.org/projects/batman-adv/wiki/Compatversion
 GLUON=(gluon
             https://github.com/freifunk-gluon/packages.git
-            8b65619f59c3bdce743c2f2fb2588fdd7079355a
-            "0001-simple-tc-Use-hotplug.d-iface-instead-of-hotplug.d-n.patch")
+            8b65619f59c3bdce743c2f2fb2588fdd7079355a)
 GLUON_PKGS="kmod-batman-adv-legacy micrond simple-tc uradvd"
 
 #official openwrt routing packages
 ROUTING=(routing
          https://git.openwrt.org/feed/routing.git
-         ea345d16a6e27c2a8fdf67bf543cc36a5f189131
-         "0002-Add-batman-adv-patch-to-remove-gw-mode-switch-messag.patch") # openwrt-18.06.2
+         ea345d16a6e27c2a8fdf67bf543cc36a5f189131) # openwrt-18.06.2
 ROUTING_PKGS="kmod-batman-adv batctl alfred babeld"
 
 FFF=(fff)
@@ -101,20 +98,17 @@  get_source() {
         if [ -n "$URL" ] && [ -n "$REV" ]; then
             checkout_git "$NAME" "$URL" "$REV"
 
-            # Patches for feeds could be stored in known directories like build_patches/$NAME/
-            # That way multiple patches for one feed could be supported
-            count=3
-            while [ "x${FEED[count]}" != "x" ]
-            do
-                local PATCH="../../../build_patches/${NAME}/${FEED[count]}"
-                if [ ! -z "$PATCH" ] ; then
-                    echo "Patching $PATCH"
-                    git -C "$NAME" am --whitespace=nowarn "$PATCH"
-                else
-                    echo "Warning, $PATCH not found."
-                fi
-                count=$(( count + 1 ))
-            done
+            # Patches for feeds are stored in known directories like feed_patches/$NAME/
+            if [ "$(find ../../feed_patches/${NAME}/*.patch | wc -l)" -ge 1 ]; then
+                for PATCH in ../../feed_patches/${NAME}/*.patch; do
+                    if [ ! -z "$PATCH" ] ; then
+                        echo "Applying $PATCH"
+                        git -C "$NAME" am --whitespace=nowarn "../$PATCH"
+                    else
+                        echo "Warning, $PATCH not found."
+                    fi
+                done
+            fi
         fi
     done
 
diff --git a/build_patches/gluon/0001-simple-tc-Use-hotplug.d-iface-instead-of-hotplug.d-n.patch b/feed_patches/gluon/0001-simple-tc-Use-hotplug.d-iface-instead-of-hotplug.d-n.patch
similarity index 100%
rename from build_patches/gluon/0001-simple-tc-Use-hotplug.d-iface-instead-of-hotplug.d-n.patch
rename to feed_patches/gluon/0001-simple-tc-Use-hotplug.d-iface-instead-of-hotplug.d-n.patch
diff --git a/build_patches/openwrt/fastd/0020-fastd_generate_key_from_urandom.patch b/feed_patches/openwrt/0020-fastd_generate_key_from_urandom.patch
similarity index 100%
rename from build_patches/openwrt/fastd/0020-fastd_generate_key_from_urandom.patch
rename to feed_patches/openwrt/0020-fastd_generate_key_from_urandom.patch
diff --git a/build_patches/routing/0002-Add-batman-adv-patch-to-remove-gw-mode-switch-messag.patch b/feed_patches/routing/0002-Add-batman-adv-patch-to-remove-gw-mode-switch-messag.patch
similarity index 100%
rename from build_patches/routing/0002-Add-batman-adv-patch-to-remove-gw-mode-switch-messag.patch
rename to feed_patches/routing/0002-Add-batman-adv-patch-to-remove-gw-mode-switch-messag.patch

Comments

Robert Langhammer April 5, 2019, 1:07 p.m.
Hallo Adrian,

find ich gut. Vor allem muss man nicht mehr im buildscript rumeditieren.

Reviewed-by: Robert Langhammer <rlanghammer@web.de>

Anmerkung unten.

Am 03.04.19 um 14:53 schrieb Adrian Schmutzler:
> The buildscript knows two different types of patches, which are
> applied to pulled-in repositories:
>
> 1. Feed patches
> Those are applied as "GIT patches" to the relevant repos,
> directly after those have been checked out.
> They reside in subfolders of the build_patches folder, and
> have to be selected individually and manually in the
> buildscript.sh.
>
> 2. Build patches
> Those are applied later in the process, just using the system
> patch tool, and changing the $target directory.
> All patches in the folder "build_patches/openwrt" are read
> and applied automatically.
>
> This is both inconsistent (two different types of patches in
> the same dir) and annoying (feed patches have to be specified
> by hand), especially for unexperienced developers.
>
> This patch addresses this by:
> - separating files into two dirs: build_patches and feed_patches
> - automatically scanning feed patches and thus having similar
>   experience for the user (I cannot think of a case where we
>   provide a patch, but do not use it)
>
> Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
> ---
>  buildscript                                        | 34 +++++++++-------------
>  ...se-hotplug.d-iface-instead-of-hotplug.d-n.patch |  0
>  .../0020-fastd_generate_key_from_urandom.patch     |  0
>  ...adv-patch-to-remove-gw-mode-switch-messag.patch |  0
>  4 files changed, 14 insertions(+), 20 deletions(-)
>  rename {build_patches => feed_patches}/gluon/0001-simple-tc-Use-hotplug.d-iface-instead-of-hotplug.d-n.patch (100%)
>  rename {build_patches/openwrt/fastd => feed_patches/openwrt}/0020-fastd_generate_key_from_urandom.patch (100%)
>  rename {build_patches => feed_patches}/routing/0002-Add-batman-adv-patch-to-remove-gw-mode-switch-messag.patch (100%)
>
> diff --git a/buildscript b/buildscript
> index 1d527c6e..a6725c5d 100755
> --- a/buildscript
> +++ b/buildscript
> @@ -23,23 +23,20 @@ PACKAGEURL="https://git.openwrt.org/feed/packages.git"
>  #official openwrt packages
>  OPENWRT=(openwrt
>           $PACKAGEURL
> -         $PACKAGEREV
> -         fastd/0020-fastd_generate_key_from_urandom.patch)
> +         $PACKAGEREV)
>  OPENWRT_PKGS="gpioctl-sysfs libugpio fastd haserl"
>  
>  ## Be careful: FFF uses COMPAT_VERSION 15 as default at the moment.
>  ## See http://www.open-mesh.org/projects/batman-adv/wiki/Compatversion
>  GLUON=(gluon
>              https://github.com/freifunk-gluon/packages.git
> -            8b65619f59c3bdce743c2f2fb2588fdd7079355a
> -            "0001-simple-tc-Use-hotplug.d-iface-instead-of-hotplug.d-n.patch")
> +            8b65619f59c3bdce743c2f2fb2588fdd7079355a)
>  GLUON_PKGS="kmod-batman-adv-legacy micrond simple-tc uradvd"
>  
>  #official openwrt routing packages
>  ROUTING=(routing
>           https://git.openwrt.org/feed/routing.git
> -         ea345d16a6e27c2a8fdf67bf543cc36a5f189131
> -         "0002-Add-batman-adv-patch-to-remove-gw-mode-switch-messag.patch") # openwrt-18.06.2
> +         ea345d16a6e27c2a8fdf67bf543cc36a5f189131) # openwrt-18.06.2
>  ROUTING_PKGS="kmod-batman-adv batctl alfred babeld"
>  
>  FFF=(fff)
> @@ -101,20 +98,17 @@ get_source() {
>          if [ -n "$URL" ] && [ -n "$REV" ]; then
>              checkout_git "$NAME" "$URL" "$REV"
>  
> -            # Patches for feeds could be stored in known directories like build_patches/$NAME/
> -            # That way multiple patches for one feed could be supported
> -            count=3
> -            while [ "x${FEED[count]}" != "x" ]
> -            do
> -                local PATCH="../../../build_patches/${NAME}/${FEED[count]}"
> -                if [ ! -z "$PATCH" ] ; then
> -                    echo "Patching $PATCH"
> -                    git -C "$NAME" am --whitespace=nowarn "$PATCH"
> -                else
> -                    echo "Warning, $PATCH not found."
> -                fi
> -                count=$(( count + 1 ))
> -            done
> +            # Patches for feeds are stored in known directories like feed_patches/$NAME/
> +            if [ "$(find ../../feed_patches/${NAME}/*.patch | wc -l)" -ge 1 ]; then

Du kannst das so lassen, es funktioniert ja. Nur so als Hinweis:

Wenn man bei einem find Ausdruck den * nicht quotet, wird der von der
Shell expandiert. Das Suchergebnis ist also vom globbing-Verhalten der
Shell abhängig. Hier könnte man also auch einfach ls nehmen. So ein
Kommando liefert den passenden Errorlevel und man kann sich das Zählen
und vergleichen sparen.

if ls ../../feed_patches/${NAME}/*.patch 2&>/dev/null ; then

Diese "unvollständigen" find finde ich auch immer etwas unschön. Wenn
schon find, dann richtig. WO soll WAS gesucht werden: find 
../../feed_patches/${NAME} -name "*.patch" . Liefert allerdings auch
Errorlevel 0, wenn nichts gefunden wird. Mann muss dann noch was
dranhängen | grep "." z.B.

> +                for PATCH in ../../feed_patches/${NAME}/*.patch; do
> +                    if [ ! -z "$PATCH" ] ; then

Man könnte das auch noch weiter eindampfen. Das obere if weglassen und
gleich

        for PATCH in $(ls ../../feed_patches/${NAME}/*.patch
2>/dev/null); do
            echo "Applying $PATCH"
            git -C "$NAME" am --whitespace=nowarn "../$PATCH"
        done

dann kann man auch den Test ob $PATCH leer ist weg lassen, da for bei
leerer Liste nicht ins do springt.

> +                        echo "Applying $PATCH"
> +                        git -C "$NAME" am --whitespace=nowarn "../$PATCH"
> +                    else
> +                        echo "Warning, $PATCH not found."

Das ist auch komisch, else kommt dran, wenn in $PATCH nix drin steht ->
"Warning, not found". Und das wird weiter oben verhindert. Kommt also
nie vor.

Robert

> +                    fi
> +                done
> +            fi
>          fi
>      done
>  
> diff --git a/build_patches/gluon/0001-simple-tc-Use-hotplug.d-iface-instead-of-hotplug.d-n.patch b/feed_patches/gluon/0001-simple-tc-Use-hotplug.d-iface-instead-of-hotplug.d-n.patch
> similarity index 100%
> rename from build_patches/gluon/0001-simple-tc-Use-hotplug.d-iface-instead-of-hotplug.d-n.patch
> rename to feed_patches/gluon/0001-simple-tc-Use-hotplug.d-iface-instead-of-hotplug.d-n.patch
> diff --git a/build_patches/openwrt/fastd/0020-fastd_generate_key_from_urandom.patch b/feed_patches/openwrt/0020-fastd_generate_key_from_urandom.patch
> similarity index 100%
> rename from build_patches/openwrt/fastd/0020-fastd_generate_key_from_urandom.patch
> rename to feed_patches/openwrt/0020-fastd_generate_key_from_urandom.patch
> diff --git a/build_patches/routing/0002-Add-batman-adv-patch-to-remove-gw-mode-switch-messag.patch b/feed_patches/routing/0002-Add-batman-adv-patch-to-remove-gw-mode-switch-messag.patch
> similarity index 100%
> rename from build_patches/routing/0002-Add-batman-adv-patch-to-remove-gw-mode-switch-messag.patch
> rename to feed_patches/routing/0002-Add-batman-adv-patch-to-remove-gw-mode-switch-messag.patch
Adrian Schmutzler April 8, 2019, 1:27 p.m.
Hallo Robert,

vielen Dank für das Feedback. Für mich ist das immer recht wertvoll, da ich ja eigtl. Linux-Noob bin, und halt das verwende, was mir gerade einfällt, aber nicht unbedingt die schöneren Alternativen kenne. (In diesem Fall war das allerdings primär ein Copy/Paste ohne viel nachzudenken).

Ich schicke eine v2 mit den Änderungen (und mache auch gleich das alte build_patches schön). Bitte dann nochmal reviewen.

Beste Grüße

Adrian

> -----Original Message-----
> From: franken-dev [mailto:franken-dev-bounces@freifunk.net] On Behalf Of
> robert
> Sent: Freitag, 5. April 2019 15:08
> To: franken-dev@freifunk.net
> Subject: Re: [PATCH] buildscript/patches: Automatically scan directories for
> feeds
> 
> Hallo Adrian,
> 
> find ich gut. Vor allem muss man nicht mehr im buildscript rumeditieren.
> 
> Reviewed-by: Robert Langhammer <rlanghammer@web.de>
> 
> Anmerkung unten.
> 
> Am 03.04.19 um 14:53 schrieb Adrian Schmutzler:
> > The buildscript knows two different types of patches, which are
> > applied to pulled-in repositories:
> >
> > 1. Feed patches
> > Those are applied as "GIT patches" to the relevant repos,
> > directly after those have been checked out.
> > They reside in subfolders of the build_patches folder, and
> > have to be selected individually and manually in the
> > buildscript.sh.
> >
> > 2. Build patches
> > Those are applied later in the process, just using the system
> > patch tool, and changing the $target directory.
> > All patches in the folder "build_patches/openwrt" are read
> > and applied automatically.
> >
> > This is both inconsistent (two different types of patches in
> > the same dir) and annoying (feed patches have to be specified
> > by hand), especially for unexperienced developers.
> >
> > This patch addresses this by:
> > - separating files into two dirs: build_patches and feed_patches
> > - automatically scanning feed patches and thus having similar
> >   experience for the user (I cannot think of a case where we
> >   provide a patch, but do not use it)
> >
> > Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
> > ---
> >  buildscript                                        | 34 +++++++++-------------
> >  ...se-hotplug.d-iface-instead-of-hotplug.d-n.patch |  0
> >  .../0020-fastd_generate_key_from_urandom.patch     |  0
> >  ...adv-patch-to-remove-gw-mode-switch-messag.patch |  0
> >  4 files changed, 14 insertions(+), 20 deletions(-)
> >  rename {build_patches => feed_patches}/gluon/0001-simple-tc-Use-
> hotplug.d-iface-instead-of-hotplug.d-n.patch (100%)
> >  rename {build_patches/openwrt/fastd => feed_patches/openwrt}/0020-
> fastd_generate_key_from_urandom.patch (100%)
> >  rename {build_patches => feed_patches}/routing/0002-Add-batman-adv-
> patch-to-remove-gw-mode-switch-messag.patch (100%)
> >
> > diff --git a/buildscript b/buildscript
> > index 1d527c6e..a6725c5d 100755
> > --- a/buildscript
> > +++ b/buildscript
> > @@ -23,23 +23,20 @@
> PACKAGEURL="https://git.openwrt.org/feed/packages.git"
> >  #official openwrt packages
> >  OPENWRT=(openwrt
> >           $PACKAGEURL
> > -         $PACKAGEREV
> > -         fastd/0020-fastd_generate_key_from_urandom.patch)
> > +         $PACKAGEREV)
> >  OPENWRT_PKGS="gpioctl-sysfs libugpio fastd haserl"
> >
> >  ## Be careful: FFF uses COMPAT_VERSION 15 as default at the moment.
> >  ## See http://www.open-mesh.org/projects/batman-
> adv/wiki/Compatversion
> >  GLUON=(gluon
> >              https://github.com/freifunk-gluon/packages.git
> > -            8b65619f59c3bdce743c2f2fb2588fdd7079355a
> > -            "0001-simple-tc-Use-hotplug.d-iface-instead-of-hotplug.d-n.patch")
> > +            8b65619f59c3bdce743c2f2fb2588fdd7079355a)
> >  GLUON_PKGS="kmod-batman-adv-legacy micrond simple-tc uradvd"
> >
> >  #official openwrt routing packages
> >  ROUTING=(routing
> >           https://git.openwrt.org/feed/routing.git
> > -         ea345d16a6e27c2a8fdf67bf543cc36a5f189131
> > -         "0002-Add-batman-adv-patch-to-remove-gw-mode-switch-
> messag.patch") # openwrt-18.06.2
> > +         ea345d16a6e27c2a8fdf67bf543cc36a5f189131) # openwrt-18.06.2
> >  ROUTING_PKGS="kmod-batman-adv batctl alfred babeld"
> >
> >  FFF=(fff)
> > @@ -101,20 +98,17 @@ get_source() {
> >          if [ -n "$URL" ] && [ -n "$REV" ]; then
> >              checkout_git "$NAME" "$URL" "$REV"
> >
> > -            # Patches for feeds could be stored in known directories like
> build_patches/$NAME/
> > -            # That way multiple patches for one feed could be supported
> > -            count=3
> > -            while [ "x${FEED[count]}" != "x" ]
> > -            do
> > -                local PATCH="../../../build_patches/${NAME}/${FEED[count]}"
> > -                if [ ! -z "$PATCH" ] ; then
> > -                    echo "Patching $PATCH"
> > -                    git -C "$NAME" am --whitespace=nowarn "$PATCH"
> > -                else
> > -                    echo "Warning, $PATCH not found."
> > -                fi
> > -                count=$(( count + 1 ))
> > -            done
> > +            # Patches for feeds are stored in known directories like
> feed_patches/$NAME/
> > +            if [ "$(find ../../feed_patches/${NAME}/*.patch | wc -l)" -ge 1 ]; then
> 
> Du kannst das so lassen, es funktioniert ja. Nur so als Hinweis:
> 
> Wenn man bei einem find Ausdruck den * nicht quotet, wird der von der
> Shell expandiert. Das Suchergebnis ist also vom globbing-Verhalten der
> Shell abhängig. Hier könnte man also auch einfach ls nehmen. So ein
> Kommando liefert den passenden Errorlevel und man kann sich das Zählen
> und vergleichen sparen.
> 
> if ls ../../feed_patches/${NAME}/*.patch 2&>/dev/null ; then
> 
> Diese "unvollständigen" find finde ich auch immer etwas unschön. Wenn
> schon find, dann richtig. WO soll WAS gesucht werden: find
> ../../feed_patches/${NAME} -name "*.patch" . Liefert allerdings auch
> Errorlevel 0, wenn nichts gefunden wird. Mann muss dann noch was
> dranhängen | grep "." z.B.
> 
> > +                for PATCH in ../../feed_patches/${NAME}/*.patch; do
> > +                    if [ ! -z "$PATCH" ] ; then
> 
> Man könnte das auch noch weiter eindampfen. Das obere if weglassen und
> gleich
> 
>         for PATCH in $(ls ../../feed_patches/${NAME}/*.patch
> 2>/dev/null); do
>             echo "Applying $PATCH"
>             git -C "$NAME" am --whitespace=nowarn "../$PATCH"
>         done
> 
> dann kann man auch den Test ob $PATCH leer ist weg lassen, da for bei
> leerer Liste nicht ins do springt.
> 
> > +                        echo "Applying $PATCH"
> > +                        git -C "$NAME" am --whitespace=nowarn "../$PATCH"
> > +                    else
> > +                        echo "Warning, $PATCH not found."
> 
> Das ist auch komisch, else kommt dran, wenn in $PATCH nix drin steht ->
> "Warning, not found". Und das wird weiter oben verhindert. Kommt also
> nie vor.
> 
> Robert
> 
> > +                    fi
> > +                done
> > +            fi
> >          fi
> >      done
> >
> > diff --git a/build_patches/gluon/0001-simple-tc-Use-hotplug.d-iface-instead-
> of-hotplug.d-n.patch b/feed_patches/gluon/0001-simple-tc-Use-hotplug.d-
> iface-instead-of-hotplug.d-n.patch
> > similarity index 100%
> > rename from build_patches/gluon/0001-simple-tc-Use-hotplug.d-iface-
> instead-of-hotplug.d-n.patch
> > rename to feed_patches/gluon/0001-simple-tc-Use-hotplug.d-iface-instead-
> of-hotplug.d-n.patch
> > diff --git a/build_patches/openwrt/fastd/0020-
> fastd_generate_key_from_urandom.patch b/feed_patches/openwrt/0020-
> fastd_generate_key_from_urandom.patch
> > similarity index 100%
> > rename from build_patches/openwrt/fastd/0020-
> fastd_generate_key_from_urandom.patch
> > rename to feed_patches/openwrt/0020-
> fastd_generate_key_from_urandom.patch
> > diff --git a/build_patches/routing/0002-Add-batman-adv-patch-to-remove-
> gw-mode-switch-messag.patch b/feed_patches/routing/0002-Add-batman-
> adv-patch-to-remove-gw-mode-switch-messag.patch
> > similarity index 100%
> > rename from build_patches/routing/0002-Add-batman-adv-patch-to-remove-
> gw-mode-switch-messag.patch
> > rename to feed_patches/routing/0002-Add-batman-adv-patch-to-remove-
> gw-mode-switch-messag.patch