1. 14 Jan, 2021 2 commits
  2. 08 Jan, 2021 1 commit
  3. 06 Dec, 2020 1 commit
    • Masahiro Yamada's avatar
      kbuild: avoid split lines in .mod files · 7d32358b
      Masahiro Yamada authored
      "xargs echo" is not a safe way to remove line breaks because the input
      may exceed the command line limit and xargs may break it up into
      multiple invocations of echo. This should never happen because
      scripts/gen_autoksyms.sh expects all undefined symbols are placed in
      the second line of .mod files.
      
      One possible way is to replace "xargs echo" with
      "sed ':x;N;$!bx;s/\n/ /g'" or something, but I rewrote the code by
      using awk because it is more readable.
      
      This issue was reported by Sami Tolvanen; in his Clang LTO patch set,
      $(multi-used-m) is no longer an ELF object, but a thin archive that
      contains LLVM bitcode files. llvm-nm prints out symbols for each
      archive member separately, which results a lot of dupications, in some
      places, beyond the system-defined limit.
      
      This problem must be fixed irrespective of LTO, and we must ensure
      zero possibility of having this issue.
      
      Link: https://lkml.org/lkml/2020/12/1/1658
      
      Reported-by: default avatarSami Tolvanen <samitolvanen@google.com>
      Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
      Reviewed-by: default avatarSami Tolvanen <samitolvanen@google.com>
      7d32358b
  4. 20 Oct, 2020 1 commit
  5. 09 Aug, 2020 3 commits
    • Masahiro Yamada's avatar
      kbuild: sort hostprogs before passing it to ifneq · 85569d19
      Masahiro Yamada authored
      
      
      The conditional:
      
        ifneq ($(hostprogs),)
      
      ... is evaluated to true if $(hostprogs) does not contain any word but
      whitespace characters.
      
        ifneq ($(strip $(hostprogs)),)
      
      ... is a safe way to avoid interpreting whitespace as a non-empty value,
      but I'd rather want to use the side-effect of $(sort ...) to do the
      equivalent.
      
      $(sort ...) is used in scripts/Makefile.host in order to drop duplication
      in $(hostprogs). It is also useful to strip excessive spaces.
      
      Move $(sort ...) before evaluating the ifneq.
      Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
      85569d19
    • Masahiro Yamada's avatar
      kbuild: move host .so build rules to scripts/gcc-plugins/Makefile · 42640b13
      Masahiro Yamada authored
      The host shared library rules are currently implemented in
      scripts/Makefile.host, but actually GCC-plugin is the only user of
      them. (The VDSO .so files are built for the target by different
      build rules) Hence, they do not need to be treewide available.
      
      Move all the relevant build rules to scripts/gcc-plugins/Makefile.
      
      I also optimized the build steps so *.so is directly built from .c
      because every upstream plugin is compiled from a single source file.
      
      I am still keeping the multi-file plugin support, which Kees Cook
      mentioned might be needed by out-of-tree plugins.
      (https://lkml.org/lkml/2019/1/11/1107
      
      )
      
      If the plugin, foo.so, is compiled from two files foo.c and foo2.c,
      then you can do like follows:
      
        foo-objs := foo.o foo2.o
      
      Single-file plugins do not need the *-objs notation.
      Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
      Acked-by: default avatarKees Cook <keescook@chromium.org>
      42640b13
    • Masahiro Yamada's avatar
      kbuild: always create directories of targets · cc8a51ca
      Masahiro Yamada authored
      
      
      Currently, the directories of objects are automatically created
      only for O= builds.
      
      It should not hurt to cater to this for in-tree builds too.
      Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
      cc8a51ca
  6. 07 Jul, 2020 1 commit
  7. 03 Jun, 2020 1 commit
  8. 01 Jun, 2020 1 commit
  9. 25 May, 2020 5 commits
  10. 17 May, 2020 1 commit
    • Masahiro Yamada's avatar
      kbuild: add infrastructure to build userspace programs · 7f3a59db
      Masahiro Yamada authored
      Kbuild supports the infrastructure to build host programs, but there
      was no support to build userspace programs for the target architecture
      (i.e. the same architecture as the kernel).
      
      Sam Ravnborg worked on this in 2014 (https://lkml.org/lkml/2014/7/13/154),
      but it was not merged. One problem at that time was, there was no good way
      to know whether $(CC) can link standalone programs. In fact, pre-built
      kernel.org toolchains [1] are often used for building the kernel, but they
      do not provide libc.
      
      Now, we can handle this cleanly because the compiler capability is
      evaluated at the Kconfig time. If $(CC) cannot link standalone programs,
      the relevant options are hidden by 'depends on CC_CAN_LINK'.
      
      The implementation just mimics scripts/Makefile.host
      
      The userspace programs are compiled with the same flags as the host
      programs. In addition, it uses -m32 or -m64 if it is found in
      $(KBUILD_CFLAGS).
      
      This new syntax has two usecases.
      
      - Sample programs
      
        Several userspace programs under samples/ include UAPI headers
        installed in usr/include. Most of them were previously built for
        the host architecture just to use the 'hostprogs' syntax.
      
        However, 'make headers' always works for the target architecture.
        This caused the arch mismatch in cross-compiling. To fix this
        distortion, sample code should be built for the target architecture.
      
      - Bpfilter
      
        net/bpfilter/Makefile compiles bpfilter_umh as the user mode helper,
        and embeds it into the kernel. Currently, it overrides HOSTCC with
        CC to use the 'hostprogs' syntax. This hack should go away.
      
      [1]: https://mirrors.edge.kernel.org/pub/tools/crosstool/
      
      Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
      Acked-by: default avatarSam Ravnborg <sam@ravnborg.org>
      7f3a59db
  11. 08 Apr, 2020 2 commits
    • Masahiro Yamada's avatar
      kbuild: link lib-y objects to vmlinux forcibly when CONFIG_MODULES=y · 7273ad2b
      Masahiro Yamada authored
      Kbuild supports not only obj-y but also lib-y to list objects linked to
      vmlinux.
      
      The difference between them is that all the objects from obj-y are
      forcibly linked to vmlinux, whereas the objects from lib-y are linked
      as needed; if there is no user of a lib-y object, it is not linked.
      
      lib-y is intended to list utility functions that may be called from all
      over the place (and may be unused at all), but it is a problem for
      EXPORT_SYMBOL(). Even if there is no call-site in the vmlinux, we need
      to keep exported symbols for the use from loadable modules.
      
      Commit 7f2084fa ("[kbuild] handle exports in lib-y objects reliably")
      worked around it by linking a dummy object, lib-ksyms.o, which contains
      references to all the symbols exported from lib.a in that directory.
      It uses the linker script command, EXTERN. Unfortunately, the meaning of
      EXTERN of ld.lld is different from that of ld.bfd. Therefore, this does
      not work with LD=ld.lld (CBL issue #515).
      
      Anyway, the build rule of lib-ksyms.o is somewhat tricky. So, I want to
      get rid of it.
      
      At first, I was thinking of accumulating lib-y objects into obj-y
      (or even replacing lib-y with obj-y entirely), but the lib-y syntax
      is used beyond the ordinary use in lib/ and arch/*/lib/.
      
      Examples:
      
       - drivers/firmware/efi/libstub/Makefile builds lib.a, which is linked
         into vmlinux in the own way (arm64), or linked to the decompressor
         (arm, x86).
      
       - arch/alpha/lib/Makefile builds lib.a which is linked not only to
         vmlinux, but also to bootloaders in arch/alpha/boot/Makefile.
      
       - arch/xtensa/boot/lib/Makefile builds lib.a for use from
         arch/xtensa/boot/boot-redboot/Makefile.
      
      One more thing, adding everything to obj-y would increase the vmlinux
      size of allnoconfig (or tinyconfig).
      
      For less impact, I tweaked the destination of lib.a at the top Makefile;
      when CONFIG_MODULES=y, lib.a goes to KBUILD_VMLINUX_OBJS, which is
      forcibly linked to vmlinux, otherwise lib.a goes to KBUILD_VMLINUX_LIBS
      as before.
      
      The size impact for normal usecases is quite small since at lease one
      symbol in every lib-y object is eventually called by someone. In case
      you are intrested, here are the figures.
      
      x86_64_defconfig:
      
         text	   data	    bss	    dec	    hex	filename
      19566602 5422072 1589328 26578002 1958c52 vmlinux.before
      19566932 5422104 1589328 26578364 1958dbc vmlinux.after
      
      The case with the biggest impact is allnoconfig + CONFIG_MODULES=y.
      
      ARCH=x86 allnoconfig + CONFIG_MODULES=y:
      
         text	   data	    bss	    dec	    hex	filename
      1175162	 254740	1220608	2650510	 28718e	vmlinux.before
      1177974	 254836	1220608	2653418	 287cea	vmlinux.after
      
      Hopefully this is still not a big deal. The per-file trimming with the
      static library is not so effective after all.
      
      If fine-grained optimization is desired, some architectures support
      CONFIG_LD_DEAD_CODE_DATA_ELIMINATION, which trims dead code per-symbol
      basis. When LTO is supported in mainline, even better optimization will
      be possible.
      
      Link: https://github.com/ClangBuiltLinux/linux/issues/515
      
      Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
      Reported-by: default avatarkbuild test robot <lkp@intel.com>
      Reviewed-by: default avatarNick Desaulniers <ndesaulniers@google.com>
      7273ad2b
    • Masahiro Yamada's avatar
      gcc-plugins: drop support for GCC <= 4.7 · 77342a02
      Masahiro Yamada authored
      Nobody was opposed to raising minimum GCC version to 4.8 [1]
      So, we will drop GCC <= 4.7 support sooner or later.
      
      We always use C++ compiler for building plugins for GCC >= 4.8.
      
      This commit drops the plugin support for GCC <= 4.7 a bit earlier,
      which allows us to dump lots of code.
      
      [1] https://lkml.org/lkml/2020/1/23/545
      
      Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
      Acked-by: default avatarKees Cook <keescook@chromium.org>
      77342a02
  12. 03 Feb, 2020 1 commit
    • Masahiro Yamada's avatar
      kbuild: rename hostprogs-y/always to hostprogs/always-y · 5f2fb52f
      Masahiro Yamada authored
      
      
      In old days, the "host-progs" syntax was used for specifying host
      programs. It was renamed to the current "hostprogs-y" in 2004.
      
      It is typically useful in scripts/Makefile because it allows Kbuild to
      selectively compile host programs based on the kernel configuration.
      
      This commit renames like follows:
      
        always       ->  always-y
        hostprogs-y  ->  hostprogs
      
      So, scripts/Makefile will look like this:
      
        always-$(CONFIG_BUILD_BIN2C) += ...
        always-$(CONFIG_KALLSYMS)    += ...
            ...
        hostprogs := $(always-y) $(always-m)
      
      I think this makes more sense because a host program is always a host
      program, irrespective of the kernel configuration. We want to specify
      which ones to compile by CONFIG options, so always-y will be handier.
      
      The "always", "hostprogs-y", "hostprogs-m" will be kept for backward
      compatibility for a while.
      Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
      5f2fb52f
  13. 06 Jan, 2020 2 commits
  14. 14 Nov, 2019 1 commit
    • Masahiro Yamada's avatar
      kbuild: remove header compile test · fcbb8461
      Masahiro Yamada authored
      There are both positive and negative options about this feature.
      At first, I thought it was a good idea, but actually Linus stated a
      negative opinion (https://lkml.org/lkml/2019/9/29/227
      
      ). I admit it
      is ugly and annoying.
      
      The baseline I'd like to keep is the compile-test of uapi headers.
      (Otherwise, kernel developers have no way to ensure the correctness
      of the exported headers.)
      
      I will maintain a small build rule in usr/include/Makefile.
      Remove the other header test functionality.
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      fcbb8461
  15. 11 Nov, 2019 2 commits
  16. 01 Oct, 2019 1 commit
  17. 06 Sep, 2019 1 commit
  18. 21 Aug, 2019 3 commits
    • Masahiro Yamada's avatar
      kbuild: move modkern_{c,a}flags to Makefile.lib from Makefile.build · eb27ea5c
      Masahiro Yamada authored
      
      
      Makefile.lib is included by Makefile.modfinal as well as Makefile.build.
      
      Move modkern_cflags to Makefile.lib in order to simplify cmd_cc_o_c
      in Makefile.modfinal. Move modkern_cflags as well for consistency.
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      eb27ea5c
    • Masahiro Yamada's avatar
      kbuild: add CONFIG_ASM_MODVERSIONS · 2ff2b7ec
      Masahiro Yamada authored
      
      
      Add CONFIG_ASM_MODVERSIONS. This allows to remove one if-conditional
      nesting in scripts/Makefile.build.
      
      scripts/Makefile.build is run every time Kbuild descends into a
      sub-directory. So, I want to avoid $(wildcard ...) evaluation
      where possible although computing $(wildcard ...) is so cheap that
      it may not make measurable performance difference.
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      Acked-by: default avatarGeert Uytterhoeven <geert@linux-m68k.org>
      2ff2b7ec
    • Masahiro Yamada's avatar
      kbuild: make single targets work more correctly · 394053f4
      Masahiro Yamada authored
      
      
      Currently, the single target build directly descends into the directory
      of the target. For example,
      
        $ make foo/bar/baz.o
      
      ... directly descends into foo/bar/.
      
      On the other hand, the normal build usually descends one directory at
      a time, i.e. descends into foo/, and then foo/bar/.
      
      This difference causes some problems.
      
      [1] miss subdir-asflags-y, subdir-ccflags-y in upper Makefiles
      
          The options in subdir-{as,cc}flags-y take effect in the current
          and its sub-directories. In other words, they are inherited
          downward. In the example above, the single target will miss
          subdir-{as,cc}flags-y if they are defined in foo/Makefile.
      
      [2] could be built in a different directory
      
          As Documentation/kbuild/modules.rst section 4.3 says, Kbuild can
          handle files that are spread over several sub-directories.
      
          The build rule of foo/bar/baz.o may not necessarily be specified in
          foo/bar/Makefile. It might be specifies in foo/Makefile as follows:
      
          [foo/Makefile]
          obj-y := bar/baz.o
      
          This often happens when a module is so big that its source files
          are divided into sub-directories.
      
          In this case, there is no Makefile in the foo/bar/ directory, yet
          the single target descends into foo/bar/, then fails due to the
          missing Makefile. You can still do 'make foo/bar/' for partial
          building, but cannot do 'make foo/bar/baz.s'. I believe the single
          target '%.s' is a useful feature for inspecting the compiler output.
      
          Some modules work around this issue by putting an empty Makefile
          in every sub-directory.
      
      This commit fixes those problems by making the single target build
      descend in the same way as the normal build does.
      
      Another change is the single target build will observe the CONFIG
      options. Previously, it allowed users to build the foo.o even when
      the corresponding CONFIG_FOO is disabled:
      
         obj-$(CONFIG_FOO) += foo.o
      
      In the new behavior, the single target build will just fail and show
      "No rule to make target ..." (or "Nothing to be done for ..." if the
      stale object already exists, but cannot be updated).
      
      The disadvantage of this commit is the build speed. Now that the
      single target build visits every directory and parses lots of
      Makefiles, it is slower than before. (But, I hope it will not be
      too slow.)
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      394053f4
  19. 14 Aug, 2019 2 commits
  20. 13 Aug, 2019 1 commit
    • Masahiro Yamada's avatar
      kbuild: use $(basename ...) for cmd_asn1_compiler · 49d5089d
      Masahiro Yamada authored
      
      
      $(basename ...) trims the last suffix. Using it is more intuitive in
      my opinion.
      
      This pattern rule makes %.asn1.c and %.asn1.h at the same time.
      Previously, the short log showed only either of them, depending on
      the target file in question.
      
      To clarify that two files are being generated by the single recipe,
      I changed the log as follows:
      
      Before:
      
        ASN.1   crypto/asymmetric_keys/x509.asn1.c
      
      After:
      
        ASN.1   crypto/asymmetric_keys/x509.asn1.[ch]
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      49d5089d
  21. 09 Aug, 2019 3 commits
    • Masahiro Yamada's avatar
      kbuild: show hint if subdir-y/m is used to visit module Makefile · c07d8d47
      Masahiro Yamada authored
      Since commit ff9b45c5
      
       ("kbuild: modpost: read modules.order instead
      of $(MODVERDIR)/*.mod"), a module is no longer built in the following
      pattern:
      
        [Makefile]
        subdir-y := some-module
      
        [some-module/Makefile]
        obj-m := some-module.o
      
      You cannot write Makefile this way in upstream because modules.order is
      not correctly generated. subdir-y is used to descend to a sub-directory
      that builds tools, device trees, etc.
      
      For external modules, the modules order does not matter. So, the
      Makefile above was known to work.
      
      I believe the Makefile should be re-written as follows:
      
        [Makefile]
        obj-m := some-module/
      
        [some-module/Makefile]
        obj-m := some-module.o
      
      However, people will have no idea if their Makefile suddenly stops
      working. In fact, I received questions from multiple people.
      
      Show a warning for a while if obj-m is specified in a Makefile visited
      by subdir-y or subdir-m.
      
      I touched the %/ rule to avoid false-positive warnings for the single
      target.
      
      Cc: Jan Kiszka <jan.kiszka@siemens.com>
      Cc: Tom Stonecypher <thomas.edwardx.stonecypher@intel.com>
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      Tested-by: default avatarJan Kiszka <jan.kiszka@siemens.com>
      c07d8d47
    • Masahiro Yamada's avatar
      kbuild: generate modules.order only in directories visited by obj-y/m · 4f2c8f30
      Masahiro Yamada authored
      
      
      The modules.order files in directories visited by the chain of obj-y
      or obj-m are merged to the upper-level ones, and become parts of the
      top-level modules.order. On the other hand, there is no need to
      generate modules.order in directories visited by subdir-y or subdir-m
      since they would become orphan anyway.
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      4f2c8f30
    • Masahiro Yamada's avatar
      kbuild: fix false-positive need-builtin calculation · d9f78edf
      Masahiro Yamada authored
      
      
      The current implementation of need-builtin is false-positive,
      for example, in the following Makefile:
      
        obj-m := foo/
        obj-y := foo/bar/
      
      ..., where foo/built-in.a is not required.
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      d9f78edf
  22. 17 Jul, 2019 4 commits
    • Masahiro Yamada's avatar
      kbuild: split out *.mod out of {single,multi}-used-m rules · 9f69a496
      Masahiro Yamada authored
      
      
      Currently, *.mod is created as a side-effect of obj-m.
      
      Split out *.mod as a dedicated build rule, which allows to unify
      the %.c -> %.o rule, and remove the single-used-m rule.
      
      This also makes the incremental build of allmodconfig faster because
      it saves $(NM) invocation when there is no change in the module.
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      9f69a496
    • Masahiro Yamada's avatar
      kbuild: remove the first line of *.mod files · 60ae1b19
      Masahiro Yamada authored
      
      
      The current format of *.mod is like this:
      
        line 1: directory path to the .ko file
        line 2: a list of objects linked into this module
        line 3: unresolved symbols (only when CONFIG_TRIM_UNUSED_KSYMS=y)
      
      Now that *.mod and *.ko are created in the same directory, the line 1
      provides no valuable information. It can be derived by replacing the
      extension .mod with .ko. In fact, nobody uses the first line any more.
      
      Cut down the first line.
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      60ae1b19
    • Masahiro Yamada's avatar
      kbuild: create *.mod with full directory path and remove MODVERDIR · b7dca6dd
      Masahiro Yamada authored
      While descending directories, Kbuild produces objects for modules,
      but do not link final *.ko files; it is done in the modpost.
      
      To keep track of modules, Kbuild creates a *.mod file in $(MODVERDIR)
      for every module it is building. Some post-processing steps read the
      necessary information from *.mod files. This avoids descending into
      directories again. This mechanism was introduced in 2003 or so.
      
      Later, commit 551559e1 ("kbuild: implement modules.order") added
      modules.order. So, we can simply read it out to know all the modules
      with directory paths. This is easier than parsing the first line of
      *.mod files.
      
      $(MODVERDIR) has a flat directory structure, that is, *.mod files
      are named only with base names. This is based on the assumption that
      the module name is unique across the tree. This assumption is really
      fragile.
      
      Stephen Rothwell reported a race condition caused by a module name
      conflict:
      
        https://lkml.org/lkml/2019/5/13/991
      
      In parallel building, two different threads could write to the same
      $(MODVERDIR)/*.mod simultaneously.
      
      Non-unique module names are the source of all kind of troubles, hence
      commit 3a48a919
      
       ("kbuild: check uniqueness of module names")
      introduced a new checker script.
      
      However, it is still fragile in the build system point of view because
      this race happens before scripts/modules-check.sh is invoked. If it
      happens again, the modpost will emit unclear error messages.
      
      To fix this issue completely, create *.mod with full directory path
      so that two threads never attempt to write to the same file.
      
      $(MODVERDIR) is no longer needed.
      
      Since modules with directory paths are listed in modules.order, Kbuild
      is still able to find *.mod files without additional descending.
      
      I also killed cmd_secanalysis; scripts/mod/sumversion.c computes MD4 hash
      for modules with MODULE_VERSION(). When CONFIG_DEBUG_SECTION_MISMATCH=y,
      it occurs not only in the modpost stage, but also during directory
      descending, where sumversion.c may parse stale *.mod files. It would emit
      'No such file or directory' warning when an object consisting a module is
      renamed, or when a single-obj module is turned into a multi-obj module or
      vice versa.
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      Acked-by: default avatarNicolas Pitre <nico@fluxnic.net>
      b7dca6dd
    • Masahiro Yamada's avatar
      kbuild: remove duplication from modules.order in sub-directories · e0e1b1ec
      Masahiro Yamada authored
      
      
      Currently, only the top-level modules.order drops duplicated entries.
      
      The modules.order files in sub-directories potentially contain
      duplication. To list out the paths of all modules, I want to use
      modules.order instead of parsing *.mod files in $(MODVERDIR).
      
      To achieve this, I want to rip off duplication from modules.order
      of external modules too.
      Signed-off-by: default avatarMasahiro Yamada <yamada.masahiro@socionext.com>
      e0e1b1ec