From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.3 (2019-12-06) on atuin X-Spam-Level: X-Spam-Status: No, score=-1.2 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS autolearn=unavailable autolearn_force=no version=3.4.3 Received: by atuin.qyliss.net (Postfix, from userid 496) id 888372287B; Sat, 30 May 2020 19:01:01 +0000 (UTC) Received: from [127.0.1.1] (localhost [IPv6:::1]) by atuin.qyliss.net (Postfix) with ESMTP id 32DA722869; Sat, 30 May 2020 19:00:52 +0000 (UTC) Received: by atuin.qyliss.net (Postfix, from userid 496) id AE0C2228B3; Sat, 30 May 2020 19:00:49 +0000 (UTC) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) by atuin.qyliss.net (Postfix) with ESMTPS id 0474322857 for ; Sat, 30 May 2020 19:00:44 +0000 (UTC) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id D32D45C00AF; Sat, 30 May 2020 15:00:42 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute7.internal (MEProxy); Sat, 30 May 2020 15:00:42 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alyssa.is; h= from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; s=fm3; bh=raae+ehjfpp+zmFJVzGwNg0/0T 0tOXlnKKB8/x3IcJI=; b=qvjRqoMuz2JF5yJagg3/QXsdCVSa0iAlaCxRm3rUVE hS4fYsJcmqjFeXUo06Lc0LeerBl62qNWScciVMkFxvTUugP1iLHEdhIQQNV4Rs/V BXUE6HW/3tjsxsA+Gxa9AH6uNW2oQOeoibMFhac8p6pKsNomghY7gNsxU3LIGuXW E7aGy+FGOUYDpBsieXlfk2htk5abmLMOKrItTUtV/hk8iZ+UeD8jZCy8KqwPcY9k 1NWJcDvjkkPh8FAyG10AviLUpJrWl42MnCO49HajxWEWYQ34FMz+Wdj+vREU3Wvx RYKZj8cd/Grvbl1L/+dDFk66Z4rgnmVOMokB+gu5jSpg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :message-id:mime-version:subject:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=raae+ehjfpp+zmFJV zGwNg0/0T0tOXlnKKB8/x3IcJI=; b=1T8Lu3rd7GJI0G6YtZgIaJiA9vEof9py5 YC+z2SkSNBplkDkvlfwHpUJ2wxT2xTzlpR20lisVC5nc6POz35QtALP0YG30iN8w 14q3EJaMcFVM/bE9dxqf1n9mJXyXDL2f7a0JIm53SdvbmwWcKz2We88zIK6HNbDi DXWQVR17S0PC7/SFxRBH3rw43zxYGNSNYyytLE4YFxrnBlBCxTUqdn6rYhaUB3Zw z9z+25wYRLhVzhpeCbpCmyhCQmicY5biX2aFIuZ8AEv9HBLpFZrl4P1xqE4i5ZtF 8Ic14jeFbNMTYWqAdqDSG9zmGS+/1CAqwG+2viJ73ICLt0+UfytNQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduhedrudeftddguddvlecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvffufffkofgggfestdekredtredttdenucfhrhhomheptehlhihsshgr ucftohhsshcuoehhihesrghlhihsshgrrdhisheqnecuggftrfgrthhtvghrnhepjeehtd ehffeffffggeevfeetffehvedujefhuefhfedttdeggfethfejtedvieffnecuffhomhgr ihhnpehgohhoghhlvghsohhurhgtvgdrtghomhenucfkphepjeelrddvfeehrdduudelrd duleegnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhep qhihlhhishhsseigvddvtddrqhihlhhishhsrdhnvght X-ME-Proxy: Received: from x220.qyliss.net (p4feb77c2.dip0.t-ipconnect.de [79.235.119.194]) by mail.messagingengine.com (Postfix) with ESMTPA id 3BE5D3280063; Sat, 30 May 2020 15:00:42 -0400 (EDT) Received: by x220.qyliss.net (Postfix, from userid 1000) id E934A3C6; Sat, 30 May 2020 19:00:40 +0000 (UTC) From: Alyssa Ross To: devel@spectrum-os.org Subject: [PATCH 1/2] chromiumOSPackages: improve update.py repo compat Date: Sat, 30 May 2020 19:00:27 +0000 Message-Id: <20200530190028.6388-1-hi@alyssa.is> X-Mailer: git-send-email 2.26.2 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Message-ID-Hash: M475DD3NFYXSKZLMICI6BXILSQ4HYDVJ X-Message-ID-Hash: M475DD3NFYXSKZLMICI6BXILSQ4HYDVJ X-MailFrom: qyliss@x220.qyliss.net X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-config-1; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; suspicious-header CC: Cole Helbling , Alyssa Ross X-Mailman-Version: 3.3.0 Precedence: list List-Id: Patches and low-level development discussion Archived-At: List-Archive: List-Help: List-Post: List-Subscribe: List-Unsubscribe: These changes make chromiumOSPackages.updateScript better understand the Repo manifest format[1]. It now properly distinguishes between paths in the Chromium OS source tree and paths on the git servers, which are usually not the same. For example, chromiumos-overlay is located in the source tree at src/third_party/chromiumos-overlay, but on chromium.googlesource.com it's located at chromiumos/overlays/chromiumos-overlays. Components are now keyed by their location in the Chromium OS source tree (and packages have all been updated for this change.) Additionally, it now understands Repo remotes. This means that it is now possible to use a Chromium OS component like Minijail, which is hosted on android.googlesource.com rather than chromiumos.googlesource.com. This was not previously possible. [1]: https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format= .md --- .../linux/chromium-os/common-mk/default.nix | 2 +- .../linux/chromium-os/crosvm/default.nix | 4 +- .../os-specific/linux/chromium-os/default.nix | 2 +- .../linux/chromium-os/libqmi/default.nix | 2 +- .../linux/chromium-os/minigbm/default.nix | 2 +- .../linux/chromium-os/modem-manager/next.nix | 2 +- .../linux/chromium-os/modp_b64/default.nix | 2 +- pkgs/os-specific/linux/chromium-os/update.py | 56 ++++++++++++------- .../linux/chromium-os/upstream-info.json | 40 ++++++------- pkgs/os-specific/linux/kernel/linux-cros.nix | 2 +- 10 files changed, 64 insertions(+), 50 deletions(-) diff --git a/pkgs/os-specific/linux/chromium-os/common-mk/default.nix b/p= kgs/os-specific/linux/chromium-os/common-mk/default.nix index f07e418b56f..73862bcaa97 100644 --- a/pkgs/os-specific/linux/chromium-os/common-mk/default.nix +++ b/pkgs/os-specific/linux/chromium-os/common-mk/default.nix @@ -24,7 +24,7 @@ , ... } @ args: =20 let - platform2 =3D fetchFromGitiles upstreamInfo.components."chromiumos/pla= tform2"; + platform2 =3D fetchFromGitiles upstreamInfo.components."src/platform2"= ; =20 attrsToGnList =3D lib.mapAttrsToList (name: value: "${name}=3D${toGn v= alue}"); =20 diff --git a/pkgs/os-specific/linux/chromium-os/crosvm/default.nix b/pkgs= /os-specific/linux/chromium-os/crosvm/default.nix index 3813e3eb75a..8698ef67a83 100644 --- a/pkgs/os-specific/linux/chromium-os/crosvm/default.nix +++ b/pkgs/os-specific/linux/chromium-os/crosvm/default.nix @@ -9,10 +9,10 @@ let else throw "no seccomp policy files available for host platform"; =20 crosvmSrc =3D fetchFromGitiles - upstreamInfo.components."chromiumos/platform/crosvm"; + upstreamInfo.components."src/platform/crosvm"; =20 adhdSrc =3D fetchFromGitiles - upstreamInfo.components."chromiumos/third_party/adhd"; + upstreamInfo.components."src/third_party/adhd"; in =20 rustPlatform.buildRustPackage rec { diff --git a/pkgs/os-specific/linux/chromium-os/default.nix b/pkgs/os-spe= cific/linux/chromium-os/default.nix index f9c6c88d072..db719e8b0f1 100644 --- a/pkgs/os-specific/linux/chromium-os/default.nix +++ b/pkgs/os-specific/linux/chromium-os/default.nix @@ -9,7 +9,7 @@ let upstreamInfo =3D lib.importJSON ./upstream-info.json; =20 chromiumos-overlay =3D (fetchFromGitiles - upstreamInfo.components."chromiumos/overlays/chromiumos-overlay") = // { + upstreamInfo.components."src/third_party/chromiumos-overlay") // { passthru.updateScript =3D ./update.py; }; =20 diff --git a/pkgs/os-specific/linux/chromium-os/libqmi/default.nix b/pkgs= /os-specific/linux/chromium-os/libqmi/default.nix index ac444848b81..ec4f44c7047 100644 --- a/pkgs/os-specific/linux/chromium-os/libqmi/default.nix +++ b/pkgs/os-specific/linux/chromium-os/libqmi/default.nix @@ -8,7 +8,7 @@ libqmi.overrideAttrs ( pname =3D "libqmi-unstable"; version =3D "2019-12-16"; =20 - src =3D fetchFromGitiles upstreamInfo.components."chromiumos/third_p= arty/libqmi"; + src =3D fetchFromGitiles upstreamInfo.components."src/third_party/li= bqmi"; =20 nativeBuildInputs =3D nativeBuildInputs ++ [ autoreconfHook autoconf-archive gtk-doc docbook-xsl-nons ]; diff --git a/pkgs/os-specific/linux/chromium-os/minigbm/default.nix b/pkg= s/os-specific/linux/chromium-os/minigbm/default.nix index 0a3cbf15997..6c532a4deb2 100644 --- a/pkgs/os-specific/linux/chromium-os/minigbm/default.nix +++ b/pkgs/os-specific/linux/chromium-os/minigbm/default.nix @@ -4,7 +4,7 @@ stdenv.mkDerivation { name =3D "minigbm"; inherit (upstreamInfo) version; =20 - src =3D fetchFromGitiles upstreamInfo.components."chromiumos/platform/= minigbm"; + src =3D fetchFromGitiles upstreamInfo.components."src/platform/minigbm= "; =20 nativeBuildInputs =3D [ pkg-config ]; buildInputs =3D [ libdrm ]; diff --git a/pkgs/os-specific/linux/chromium-os/modem-manager/next.nix b/= pkgs/os-specific/linux/chromium-os/modem-manager/next.nix index a074e82ff4b..91aa18b3846 100644 --- a/pkgs/os-specific/linux/chromium-os/modem-manager/next.nix +++ b/pkgs/os-specific/linux/chromium-os/modem-manager/next.nix @@ -9,7 +9,7 @@ modemmanager.overrideAttrs ( version =3D "2019-10-17"; =20 src =3D fetchFromGitiles - upstreamInfo.components."chromiumos/third_party/modemmanager-next"= ; + upstreamInfo.components."src/third_party/modemmanager-next"; =20 nativeBuildInputs =3D nativeBuildInputs ++ [ autoreconfHook libqmi l= ibxslt ]; =20 diff --git a/pkgs/os-specific/linux/chromium-os/modp_b64/default.nix b/pk= gs/os-specific/linux/chromium-os/modp_b64/default.nix index 50d91479193..557769336ce 100644 --- a/pkgs/os-specific/linux/chromium-os/modp_b64/default.nix +++ b/pkgs/os-specific/linux/chromium-os/modp_b64/default.nix @@ -3,7 +3,7 @@ common-mk { platformSubdir =3D "modp_b64"; =20 - src =3D fetchFromGitiles upstreamInfo.components."aosp/platform/extern= al/modp_b64"; + src =3D fetchFromGitiles upstreamInfo.components."src/third_party/modp= _b64"; =20 installPhase =3D '' mkdir -p $out/lib diff --git a/pkgs/os-specific/linux/chromium-os/update.py b/pkgs/os-speci= fic/linux/chromium-os/update.py index 1405c534d8b..58a15dbb8e8 100755 --- a/pkgs/os-specific/linux/chromium-os/update.py +++ b/pkgs/os-specific/linux/chromium-os/update.py @@ -13,16 +13,16 @@ from re import MULTILINE, fullmatch, match, search from urllib.request import urlopen =20 # ChromiumOS components used in Nixpkgs -components =3D [ - 'aosp/platform/external/modp_b64', - 'chromiumos/overlays/chromiumos-overlay', - 'chromiumos/platform/crosvm', - 'chromiumos/platform/minigbm', - 'chromiumos/platform2', - 'chromiumos/third_party/adhd', - 'chromiumos/third_party/kernel', - 'chromiumos/third_party/libqmi', - 'chromiumos/third_party/modemmanager-next', +component_paths =3D [ + 'src/platform/crosvm', + 'src/platform/minigbm', + 'src/platform2', + 'src/third_party/adhd', + 'src/third_party/chromiumos-overlay', + 'src/third_party/kernel/v5.4', + 'src/third_party/libqmi', + 'src/third_party/modemmanager-next', + 'src/third_party/modp_b64', ] =20 git_root =3D 'https://chromium.googlesource.com/' @@ -65,14 +65,28 @@ with urlopen(f'{buildspecs_url}{chrome_major_version}= /?format=3DTEXT') as resp: buildspecs.sort(reverse=3DTrue) buildspec =3D splitext(buildspecs[0])[0] =20 -revisions =3D {} +components =3D {} =20 # Read the buildspec, and extract the git revisions for each component. with urlopen(f'{buildspecs_url}{chrome_major_version}/{buildspec}.xml?fo= rmat=3DTEXT') as resp: xml =3D base64.decodebytes(resp.read()) root =3D etree.fromstring(xml) + + default_remote =3D root.find('default').get('remote') + remotes =3D {} + for remote in root.findall('remote'): + remotes[remote.get('name')] =3D remote.get('fetch') + for project in root.findall('project'): - revisions[project.get('name')] =3D project.get('revision') + name =3D project.get('name') + path =3D project.get('path') + remote_name =3D project.get('remote') or default_remote + remote =3D remotes[remote_name] + + components[path] =3D { + 'revision': project.get('revision'), + 'url': f'{remote}/{name}', + } =20 # Initialize the data that will be output from this script. Leave the # rc number in buildspec so nobody else is subject to the same level @@ -84,15 +98,15 @@ paths =3D {} # Fill in the 'components' dictionary with the output from # nix-prefetch-git, which can be passed straight to fetchGit when # imported by Nix. -for component in components: - name =3D component.split('/')[-1] - url =3D f'{git_root}{component}' - rev =3D revisions[component] +for path in component_paths: + name =3D path.split('/')[-1] + url =3D components[path]['url'] + rev =3D components[path]['revision'] tarball =3D f'{url}/+archive/{rev}.tar.gz' output =3D subprocess.check_output(['nix-prefetch-url', '--print-pat= h', '--unpack', '--name', name, tarball]) - (sha256, path) =3D output.decode('utf-8').splitlines() - paths[component] =3D path - data['components'][component] =3D { + (sha256, store_path) =3D output.decode('utf-8').splitlines() + paths[path] =3D store_path + data['components'][path] =3D { 'name': name, 'url': url, 'rev': rev, @@ -100,14 +114,14 @@ for component in components: } =20 # Get the version number of the kernel. -kernel =3D paths['chromiumos/third_party/kernel'] +kernel =3D paths['src/third_party/kernel/v5.4'] makefile =3D open(f'{kernel}/Makefile').read() version =3D search(r'^VERSION =3D (.+)$', makefile, MULTILINE)[1] patchlevel =3D search(r'^PATCHLEVEL =3D (.*?)$', makefile, MULTILINE)[1] sublevel =3D search(r'^SUBLEVEL =3D (.*?)$', makefile, MULTILINE)[1] extra =3D search(r'^EXTRAVERSION =3D[ \t]*(.*?)$', makefile, MULTILINE)[= 1] full_ver =3D '.'.join(filter(None, [version, patchlevel, sublevel])) + e= xtra -data['components']['chromiumos/third_party/kernel']['version'] =3D full_= ver +data['components']['src/third_party/kernel/v5.4']['version'] =3D full_ve= r =20 # Finally, write the output. with open(dirname(__file__) + '/upstream-info.json', 'w') as out: diff --git a/pkgs/os-specific/linux/chromium-os/upstream-info.json b/pkgs= /os-specific/linux/chromium-os/upstream-info.json index 5264a72a6c9..842912800bc 100644 --- a/pkgs/os-specific/linux/chromium-os/upstream-info.json +++ b/pkgs/os-specific/linux/chromium-os/upstream-info.json @@ -1,60 +1,60 @@ { "version": "81.12871.0.0-rc1", "components": { - "aosp/platform/external/modp_b64": { - "name": "modp_b64", - "url": "https://chromium.googlesource.com/aosp/platform/external/m= odp_b64", - "rev": "ae26ad8516036774fd3b6e84a98a89955dcf5f61", - "sha256": "03ja5fkq3wk0qcnfnqcj09lciky6n6zdj544xs1jfssjhz9yj8hi" - }, - "chromiumos/overlays/chromiumos-overlay": { - "name": "chromiumos-overlay", - "url": "https://chromium.googlesource.com/chromiumos/overlays/chro= miumos-overlay", - "rev": "be285b729ef2af700cc248f741af20c387b5a1bc", - "sha256": "18y2icdb3hxqxxaaz8sylw21q2qwcpjyns75ig7ahc4vc1k5hg7s" - }, - "chromiumos/platform/crosvm": { + "src/platform/crosvm": { "name": "crosvm", "url": "https://chromium.googlesource.com/chromiumos/platform/cros= vm", "rev": "8b8c01e1ad31718932491e4aee63f56109a138e2", "sha256": "1qmf1k06pwynh15c3nr9m6v90z2pkk930xniwvlvbvnazrk4rllg" }, - "chromiumos/platform/minigbm": { + "src/platform/minigbm": { "name": "minigbm", "url": "https://chromium.googlesource.com/chromiumos/platform/mini= gbm", "rev": "fbfb338e7db246f05ae68da97141fbd49481584b", "sha256": "11p6v4nmqivyqyc8lmypbwqv9q819lhx6v3igfsv3455mmq02kcb" }, - "chromiumos/platform2": { + "src/platform2": { "name": "platform2", "url": "https://chromium.googlesource.com/chromiumos/platform2", "rev": "6e0d88d0bea8e7cb28a310298cc8c77ed742f60a", "sha256": "13vrbnl3ck8cxiyifc58cs9i0yby92vp5si27ihrx33v6bdlir4s" }, - "chromiumos/third_party/adhd": { + "src/third_party/adhd": { "name": "adhd", "url": "https://chromium.googlesource.com/chromiumos/third_party/a= dhd", "rev": "f361d5b02623274723bff251dafa1e2a2887b013", "sha256": "1p8iwjwgmcgmzri03ik2jaid8l0ch0bzn6z9z64dix1hlrvrlliw" }, - "chromiumos/third_party/kernel": { - "name": "kernel", + "src/third_party/chromiumos-overlay": { + "name": "chromiumos-overlay", + "url": "https://chromium.googlesource.com/chromiumos/overlays/chro= miumos-overlay", + "rev": "be285b729ef2af700cc248f741af20c387b5a1bc", + "sha256": "18y2icdb3hxqxxaaz8sylw21q2qwcpjyns75ig7ahc4vc1k5hg7s" + }, + "src/third_party/kernel/v5.4": { + "name": "v5.4", "url": "https://chromium.googlesource.com/chromiumos/third_party/k= ernel", "rev": "17611b6e636c8ccae25e873e659d69c20b215815", "sha256": "0an8w3660v2zf2crb68ambg7vzw2rajpjmcby1am08qy37lqsfkd", "version": "5.4.16" }, - "chromiumos/third_party/libqmi": { + "src/third_party/libqmi": { "name": "libqmi", "url": "https://chromium.googlesource.com/chromiumos/third_party/l= ibqmi", "rev": "6267c12fef0be9a95d687bf4b476593eeffbaaf1", "sha256": "00yjjcngjkbgxmmf79ciddk9yq393a94230ahq8vszyfygzhs3rz" }, - "chromiumos/third_party/modemmanager-next": { + "src/third_party/modemmanager-next": { "name": "modemmanager-next", "url": "https://chromium.googlesource.com/chromiumos/third_party/m= odemmanager-next", "rev": "a4d3a4de1fdab7c282f1a188d2d3d0aaa24f0698", "sha256": "1b1l1jfiyl7fnm29pp9npi0gdlqcw1jmmbigf0qlndqdh0ly2izg" + }, + "src/third_party/modp_b64": { + "name": "modp_b64", + "url": "https://chromium.googlesource.com/aosp/platform/external/m= odp_b64", + "rev": "ae26ad8516036774fd3b6e84a98a89955dcf5f61", + "sha256": "03ja5fkq3wk0qcnfnqcj09lciky6n6zdj544xs1jfssjhz9yj8hi" } } } diff --git a/pkgs/os-specific/linux/kernel/linux-cros.nix b/pkgs/os-speci= fic/linux/kernel/linux-cros.nix index 6a4a5771b38..789a9fb6fd2 100644 --- a/pkgs/os-specific/linux/kernel/linux-cros.nix +++ b/pkgs/os-specific/linux/kernel/linux-cros.nix @@ -3,7 +3,7 @@ , ... } @ args: =20 let - versionData =3D upstreamInfo.components."chromiumos/third_party/kernel= "; + versionData =3D upstreamInfo.components."src/third_party/kernel/v5.4"; in =20 with lib; --=20 2.26.2