Bug 4203

Summary: buildSRPMFromSCM fail with old sources , need run rfpkg new-sources before to work.
Product: Fedora Reporter: Sérgio Basto <sergio>
Component: rfpkg-minimalAssignee: Ben Rosser <rosser.bjr>
Status: RESOLVED FIXED    
Severity: minor CC: kwizart, leigh123linux, lxtnow, matthias, trpost
Priority: P5    
Version: unspecified   
Hardware: All   
OS: GNU/Linux   
namespace:

Description Sérgio Basto 2016-08-18 03:37:20 CEST
Hi, when I ask one build and koji do buildSRPMFromSCM 

Example:  
http://koji.rpmfusion.org/koji/taskinfo?taskID=22292

fails with: 
DEBUG util.py:495:  Executing command: ['rfpkg-minimal', 'sources'] with env {'HOSTNAME': 'mock', 'TERM': 'vt100', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'http_proxy': 'http://proxy:3128', 'SHELL': '/bin/bash', 'HOME': '/builddir', 'PROMPT_COMMAND': 'printf "\x1b]0;<mock-chroot>\x07<mock-chroot>"', 'LANG': 'en_US.UTF-8'} and shell False
(...)
DEBUG util.py:417:  curl: (22) The requested URL returned error: 404 Not Found
DEBUG util.py:542:  Child return code was: 22

after I did 
rfpkg srpm 
rfpkg new-sources twolame-0.3.13.tar.gz

and a buildSRPMFromSCM works with : DEBUG util.py:495:  Executing command: ['rfpkg-minimal', 'sources'] with env {'HOME': '/builddir', 'LANG': 'en_US.UTF-8', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'PROMPT_COMMAND': 'printf "\x1b]0;<mock-chroot>\x07<mock-chroot>"', 'http_proxy': 'http://proxy:3128', 'HOSTNAME': 'mock', 'TERM': 'vt100', 'SHELL': '/bin/bash'} and shell False
(...)
DEBUG util.py:417:  twolame-0.3.13.tar.gz: OK

for example: http://koji.rpmfusion.org/koji/taskinfo?taskID=22317

it is not an critical bug ...
Comment 1 leigh scott 2016-08-18 14:43:23 CEST
I get the same when running 'rfpkg srpm' locally, looks like there is a issue with the lookaside cache.
Comment 2 Nicolas Chauvet 2016-08-18 14:56:28 CEST
There is no symlink with the old path so rfpkg-minimal tries to use the newer path layout (using md5) whereas there is no hash indication with the old path.

Re-assigned to rfpkg-minimal owner since the infra has switched to instead over rfpkg and the lot of dependencies.
Comment 3 A. Trande (sagitter) 2016-08-19 18:13:27 CEST
I have not idea about how to fix this issue.
Suggestions?
Comment 4 Sérgio Basto 2016-08-19 18:22:31 CEST
(In reply to comment #3)
> I have not idea about how to fix this issue.
> Suggestions?

kwizart said :  
There is no symlink with the old path so rfpkg-minimal tries to use the newer
path layout (using md5) whereas there is no hash indication with the old path.


we miss a symlink somewhere
Comment 5 A. Trande (sagitter) 2016-08-19 18:59:30 CEST
(In reply to comment #4)
> (In reply to comment #3)
> > I have not idea about how to fix this issue.
> > Suggestions?
> 
> kwizart said :  
> There is no symlink with the old path so rfpkg-minimal tries to use the newer
> path layout (using md5) whereas there is no hash indication with the old path.
> 
> 
> we miss a symlink somewhere

Can you try with this release?
http://koji.rpmfusion.org/koji/taskinfo?taskID=22994
Comment 6 Sérgio Basto 2016-08-19 19:11:35 CEST
(In reply to comment #5)
> (In reply to comment #4)
> > (In reply to comment #3)
> > > I have not idea about how to fix this issue.
> > > Suggestions?
> > 
> > kwizart said :  
> > There is no symlink with the old path so rfpkg-minimal tries to use the newer
> > path layout (using md5) whereas there is no hash indication with the old path.
> > 
> > 
> > we miss a symlink somewhere
> 
> Can you try with this release?
> http://koji.rpmfusion.org/koji/taskinfo?taskID=22994

the problem is in rfpkg-mininal , not in rfpkg, IIUC .
Comment 7 A. Trande (sagitter) 2016-08-19 19:14:29 CEST
(In reply to comment #6)
> (In reply to comment #5)
> > (In reply to comment #4)
> > > (In reply to comment #3)
> > > > I have not idea about how to fix this issue.
> > > > Suggestions?
> > > 
> > > kwizart said :  
> > > There is no symlink with the old path so rfpkg-minimal tries to use the newer
> > > path layout (using md5) whereas there is no hash indication with the old path.
> > > 
> > > 
> > > we miss a symlink somewhere
> > 
> > Can you try with this release?
> > http://koji.rpmfusion.org/koji/taskinfo?taskID=22994
> 
> the problem is in rfpkg-mininal , not in rfpkg, IIUC .

I'm not the maintainer of rfpkg-minimal. :)
Ben, it's your time.
Comment 8 Ben Rosser 2016-08-19 20:31:57 CEST
> There is no symlink with the old path so rfpkg-minimal tries to use the newer
path layout (using md5) whereas there is no hash indication with the old path.

While this is what rfpkg-minimal is doing, as far as I can tell, it succeeds, because the .../md5/hash/... URLs redirect to .../hash/... (or possibly, the other way around).

* The linked Koji build succeeded.

* I cloned the twolame sources and ran rfpkg-minimal (using 0.2.0) and the file downloaded successfully.

* I also could not reproduce this issue using an older rfpkg-minimal (which is still the release in the stable F24 repository), or the rfpkg currently in the repositories.

* The following URL (which is where rfpkg-minimal 0.2.0 tries to download from in this case) appears to be valid: http://pkgs.rpmfusion.org/repo/pkgs/free/twolame/twolame-0.3.13.tar.gz/md5/4113d8aa80194459b45b83d4dbde8ddb/twolame-0.3.13.tar.gz

So perhaps I'm missing something, but... I am not seeing an issue in rfpkg-minimal here. Can someone provide detailed instructions on how to reproduce whatever the problem is?
Comment 9 Sérgio Basto 2016-08-20 13:15:05 CEST
(In reply to comment #8)
> I am not seeing an issue in
> rfpkg-minimal here. Can someone provide detailed instructions on how to
> reproduce whatever the problem is?

After we run rfpkg new-sources foo package, the issue is fixed , I found one that is not fixed yet fixed (z-push), so we may replicate the problem with: 

rfpkg clone free/z-push
cd z-push/

rfpkg-minimal srpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (22) The requested URL returned error: 404 Not Found

while: 

rfpkg srpm
Downloading z-push-2.2.8.tar.gz
######################################################################## 100.0%


Wrote: /home/sergio/rpmfusion/new/z-push/z-push-2.2.8-2.fc26.src.rpm
Comment 10 Ben Rosser 2016-08-20 20:05:54 CEST
Oh, now I understand, thanks!

Hm. I guess that means we should just try to download from the other URL if the .../md5/hash/... URL fails. I'll implement that and push out rfpkg-minimal 0.2.1.
Comment 11 Ben Rosser 2016-08-20 20:31:11 CEST
Okay; I believe I've fixed this. Can you grab rfpkg-minimal from git (https://raw.githubusercontent.com/rpmfusion-infra/rfpkg-minimal/4312ee113d3aa486bee8e31c639a2d0ac8119d5f/bin/rfpkg-minimal) and verify this works now, before I build 0.2.2?

With this version of the script, I'm able to succesfully run rfpkg-minimal sources in git checkouts for both rfpkg-minimal and z-push successfully.
Comment 12 Sérgio Basto 2016-08-21 02:41:02 CEST
It worked but with lots of logs ... , but I don't have opinion , let me know if it finished .

Thanks.
Comment 13 Ben Rosser 2016-08-21 02:52:29 CEST
Yes, unfortunately you'll see curl "fail" to download from the first URL before trying the second and succeeding, hence the noise. It'll do this for multiple files in the sources file too.

Anyway, good to know, thanks for testing. :) I'll tag and build 0.2.1 then.
Comment 14 Ben Rosser 2016-08-21 03:36:59 CEST
I have pushed and built rfpkg-minimal-0.2.1 for F23 and F24. Builds on F25 and master seem to be failing due to some unrelated shenanigans?

<TC01> builds appear to be failing at the moment on Rawhide and F25: I'm getting lots of "DEBUG util.py:421:  http://dl.fedoraproject.org/pub/fedora/linux/development/25/Everything/x86_64/os/Packages/c/curl-7.50.1-1.fc25.x86_64.rpm: [Errno -1] Package does not match intended download. Suggestion: run yum --enablerepo=build clean metadata and try again"
<TC01> e.g. http://koji.rpmfusion.org/kojifiles/work/tasks/3518/23518/root.log

(I'm guessing there were just F25 and Rawhide pushes or something. I'll try again in a bit).
Comment 15 Sérgio Basto 2016-08-21 03:48:09 CEST
yes, F25 builds are failing since yesterday, I don't know if related [1] but all compose, on Fedora proper, are failing too . 


[1] 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/5TAGFSEZXHO6KDUQELCXKKNH5NP5LHP2/
Comment 16 Sérgio Basto 2016-08-23 00:02:03 CEST
(In reply to comment #14)
> I have pushed and built rfpkg-minimal-0.2.1 for F23 and F24. Builds on F25 and
> master seem to be failing due to some unrelated shenanigans?


rfpkg-minimal-0.2.1-1.fc25 and rfpkg-minimal-0.2.1-1.fc26 built , closing .

Many thanks.
Comment 17 Sérgio Basto 2016-08-23 21:46:28 CEST
Fixed, http://koji.rpmfusion.org/kojifiles/work/tasks/5208/25208/root.log proves it . 

Thanks.